How benchmarks work

A benchmark result answers one question: on this workload, how fast, how much, and how cheaply does this model run on this GPU with this engine setup? Every result is produced by a recipe, and that recipe is what you deploy.

The three measurements

Measurement	What it answers	Reported as
Speed	How fast does one request feel?	Time to first token (p50, p99), output tokens per second per request, inter-token latency
Throughput	How much work does one GPU do?	Output tokens per second per GPU, and input tokens per second per GPU, at the latency target
Cost	What does a token cost to produce?	USD per million output tokens and per million input tokens, plus a blended figure for the workload's input/output mix

Speed

Measured per request, from the client side, over a streaming connection:

• Time to first token (TTFT): from sending the request to receiving the first output token.
• Output speed: output tokens divided by the time from first to last token.
• Inter-token latency (ITL): the gap between consecutive output tokens.

Reported at p50 and p99 for each concurrency level, because speed falls as more requests share a GPU.

Throughput

Concurrency is raised step by step. At each step, throughput is the total output tokens per second across all requests, divided by the number of GPUs the recipe uses (for a disaggregated recipe, prefill and decode GPUs together).

The headline number is throughput at the latency target: the highest throughput reached while p99 TTFT and p99 ITL stay within the target you choose. A setup that is fast at one request but collapses under load scores low here.

Cost

cost per million output tokens = GPU price per hour ÷ (output tokens per second per GPU × 3600) × 1,000,000

• GPU price per hour is the provider's public on-demand list price for that GPU on the run date, per GPU. Each result records the price and the provider it came from.
• Throughput is the throughput at the latency target, so cost reflects a deployment that meets the target, not one pushed past it.
• Input cost is computed the same way from input tokens per second per GPU.
• Blended cost weights input and output by the workload's measured mix.

On your own hardware, reserved capacity or committed spend, your price is different. denv bench run uses your fleet's price, and denv bench get shows the price any published result assumed.

Workloads

Speed and throughput depend on the shape of the traffic, so results are reported per workload:

Workload	Shape
chat	Short prompts, medium answers, many concurrent users
agent	Long, growing conversations with tool calls and a large shared prefix
rag	Long prompts with retrieved context, short answers
coding	Long prompts with code context, long answers, strict latency
batch	Throughput first, latency relaxed

A workload is defined by input and output lengths, arrival timing and prefix reuse. Shapes and prompts are published.

What a result contains

Field	Meaning
Recipe	Model, engine version, hardware, topology, arguments and API adapter, pinned by version
Workload and target	The traffic shape and the latency target used for throughput and cost
Speed	TTFT, output speed and ITL at p50 and p99, per concurrency level
Throughput	Tokens per second per GPU per concurrency level, and at the target
Cost	Per million output, input and blended tokens, with the GPU price and source used
Provenance	Date, GPU count, engine and driver versions, run id, and the command to reproduce it

Ranking

denv bench ranks results that meet your latency target by --rank cost (default), --rank throughput, or --rank speed. Results that miss the target are listed after, marked.

Reproducing a result

denv bench get bench_7Qk2
denv bench run --recipe deepseek-v4.1-flash/deepinfer@3 --target my-fleet --workload agent

On your own GPUs with your own traffic shape and price:

denv bench run --recipe deepseek-v4.1-flash/deepinfer@3 --target my-fleet --traffic shape.jsonl

shape.jsonl holds lengths and timing per request, not contents.

What benchmarks do not measure

Benchmarks measure speed, throughput and cost, not output quality. Quality is checked when you deploy: golden prompts and parity with the model's official API run as gates before a release takes traffic.

Independence

denv does not sell GPUs or tokens, and recipes run on published engines. Methods, workload shapes and recipes are public, and any result can be reproduced with one command.

From result to deployment

denv deploy --from-bench bench_7Qk2 --target aws-prod

This creates a release from the result's recipe, so what runs is what was measured. From there it is an ordinary release: it goes through stages, can be optimized on your traffic, and rolls back on its own.