Key takeaways

  1. The recorded run contains 120,000 requests on Inferentia2 and NVIDIA A10G. It is not a Trainium2 result.
  2. The per request latency distributions and error counts are usable. At 128 concurrent requests the median was 178.98 milliseconds on Inferentia2 and 991.49 milliseconds on A10G.
  3. The throughput formula did not measure elapsed wall time. Every cost and fleet claim derived from it was withdrawn.

The first version of this benchmark had an impressive answer: lower latency, higher throughput and lower cost on purpose built silicon. The latency result survived review. The economics did not. The harness had divided the number of successful requests by the latency of the single slowest request, not by the elapsed time of the run.

CorrectionThe public resume previously described these measurements as Trainium2 results and quoted a cost advantage. The run was performed on Inferentia2. The labels were changed later without a new hardware run. The throughput and cost claims have been removed.

What was measured

The workload was DistilBERT sentiment classification with 67 million parameters and a maximum sequence length of 128. Each target received 10,000 requests at six concurrency levels: 1, 8, 32, 64, 128 and 256.

The measured targets were AWS Inferentia2 on inf2.xlarge and NVIDIA A10G on g5.xlarge. Together the result files contain 120,000 requests.

Concurrent requestsInferentia2 p50A10G p50Inferentia2 p99A10G p99Errors Inf2 / A10G
12.24 ms4.63 ms2.41 ms5.42 ms0 / 0
810.47 ms67.16 ms17.44 ms79.68 ms0 / 0
3244.28 ms244.15 ms52.87 ms307.98 ms0 / 0
6489.24 ms491.49 ms99.43 ms645.18 ms0 / 0
128178.98 ms991.49 ms188.92 ms1188.46 ms0 / 0
256185.87 ms1467.65 ms3699.71 ms5916.76 ms0 / 29

Download the sanitized audited result fields.

Latency at 128 concurrent requestsInferentia2 median latency was 178.98 milliseconds and A10G median latency was 991.49 milliseconds. The p50 to p99 spread was about 10 milliseconds on Inferentia2 and 197 milliseconds on A10G. 128 CONCURRENT REQUESTS Median latency Inferentia2178.98 ms A10G991.49 ms Tail spread: p99 minus p50 INFERENTIA29.94 ms A10G196.97 ms The chart reports measured request latency. It does not report sustained throughput.
Figure 1. The valid result is a latency distribution. Median latency was 5.54 times lower on Inferentia2 at this concurrency level, with a much tighter tail.

The economics claim failed

Throughput is a rate. It needs the number of successful requests and the elapsed time required to complete them. The harness had all individual request latencies but never recorded the start and end of the full request set.

Instead, it used the largest individual latency as if that were the duration of the run.

Wrong rate: T = N ÷ max(Lᵢ)
Required rate: T = N success ÷ (t end minus t start)

In the first expression, N is the number of successful requests and max(Lᵢ) is the slowest single request. In the required expression, t start and t end bound the complete run. Concurrent requests overlap, so one request latency cannot stand in for total elapsed time.

The cost per million calculation used that invalid throughput. Fleet sizing then used the cost result. One measurement defect therefore propagated into every economic conclusion.

How one measurement defect propagatedSlowest request latency was used as wall time, creating invalid throughput, cost and fleet claims. DEFECT PROPAGATION Slowest requestused as wall time Invalidthroughput Invalid costper million Invalid fleeteconomics Every metric below the measurement inherits its defect.
Figure 2. Derived metrics inherit the validity of the measurement beneath them.

What survived the audit

The request count survived. The result files contain 10,000 requests for each of six concurrency levels on each target.

The per request latency distributions survived, and the error counts survived: zero on Inferentia2 at 256 concurrent requests against 29 on A10G. These are observed results under this harness, model and serving setup, not universal hardware claims.

What was withdrawn

  1. Trainium2 attribution: the recorded result was produced on Inferentia2. Renaming the profile and rewriting the server after a demonstration did not create a new measurement.
  2. Throughput: the harness did not record elapsed run time.
  3. Cost per million: the calculation depended on invalid throughput.
  4. Fleet savings: the projection inherited the same defect.
  5. Architectural cause: the run observed different behavior but did not isolate which hardware or software mechanism caused it.
A number that the harness did not measure is not made true by a better chart.

How the rerun should be designed

A credible economics run starts the clock immediately before the request set is submitted and stops it after every future completes. It reports successful requests divided by that elapsed wall time.

The hardware identity and price must come from the actual run. The serving stack must be comparable on both targets, or the limitation must be stated. Multiple independent trials should report variation rather than one result. Accuracy on the target data should be checked after compilation.

Latency, throughput and economics answer different questions. A valid latency result does not permit an economic conclusion. The benchmark still produced useful evidence. Its most useful result may be the audit that narrowed the claim to what the experiment actually measured.

Kaushal Dave

Kaushal Dave

Kaushal Dave leads Applied AI from strategy to production for global enterprises. His work spans Forward Deployed Engineering, supply chain, global retail pricing and economic decisions.

ProfileLinkedIn