50.8 to 213 tokens per second on one H100

On a Friday night I ran the first real demo: Qwen3.8-27B on one H100 in a Nebius account, called from the ordinary OpenAI client. It worked. It did 50.8 tokens a second.

By the end of the night the same kind of GPU did 213. Same open engine, vLLM. No new hardware and, that night, no custom kernel. This is the path, adapted from the X article. The longer explanation of speculative decoding, with the same 50 / 164 / 213 ladder, is the thread next to it.

If you want the 23-provider race, that is a different measurement on a later day. The H100’s median there was 241 tok/s, not 213.

The ceiling

With one request in flight, every new token reads the weights once. Qwen3.8-27B in bf16 is 55.6 GB. An H100’s memory bandwidth is about 3.35 TB/s. Divide, and the best case is about 60 tokens a second. I was at 50.8, already about 82% of that ceiling. Flags were not going to reach 200.

Two levers are left. Read fewer bytes per step, or produce more than one token per step. I used both.

What changed

Qwen publishes an FP8 checkpoint. Eight-bit weights are half of bf16, so each step reads half as much and the ceiling roughly doubles.

The model also ships a small multi-token prediction head. Speculative decoding with that head drafting 3 tokens at a time reached 164 tok/s. The full model checks the drafts in one pass. A draft it keeps is a token that did not need another full read of the weights.

A 4-bit build with the same head reached 179. I did not use it. Four-bit weights stop being the FP8 model’s answers. I would rather be slower and know which model I am serving.

DFlash2 is an external drafter for this checkpoint. It proposes a block of tokens in one pass instead of guessing one by one. Drafting 7 tokens, on FP8 weights, reached 213 tok/s. The big model still verifies every drafted token and keeps only what it would have written. The drafter changes when the answer arrives. It does not change the answer. The model is 27.8 billion parameters, about 31 GB in FP8. DFlash2 is about 1.9 billion parameters and 3.8 GB.

The day was 50.8, then 164, then 179 rejected, then 213.

A smaller race the same night

Six prompts: code, writing, a summary, a comparison, an email. Four rounds, so 24 requests per provider, one at a time, up to 512 tokens, thinking off, the same sampling settings. Providers pinned through OpenRouter with fallbacks off. I called my endpoint directly.

Median output speed:

Who	Median tok/s
Cerebras, wafer-scale chip	1,092
One H100, FP8 + DFlash2	213
ModelRun, GPU, fp4	191
Modal, GPU	125
Parasail, GPU, fp8	73

Cerebras is not a GPU. Among the GPU rows that night, the H100 was the fastest, and its first token was the tightest I saw: 221 ms median, 234 ms at the 90th percentile. OpenRouter adds a hop to everyone else’s first token. It does not add that hop to their output speed.

Load, and a cold start

One request at a time flatters a dedicated GPU. On ordinary prompts, one request in flight did 196 tok/s. Sixteen in flight did 130 tok/s each, and the GPU as a whole did 1,817 tok/s. First token was about 0.3 seconds. Speculation helps most when the card has spare compute. As the batch fills, each request slows down and the total goes up. 1,817 is the number that decides whether the GPU pays for itself.

At one request at a time, an H100 at $4.50 an hour doing 213 tok/s is about $5.87 per million output tokens. That was the rental I was on, not a denv price. denv does not sell GPU hours. A dedicated GPU only makes sense with steady traffic, which is why the 1,817 figure matters more than the 213.

A cold start on a fresh machine took 12 minutes 46 seconds: on the order of 4 minutes for the image, 4 for the weights, 5 to start the engine. With weights on disk, a restart took 99 seconds. The breakdown I posted that day was image pull 3m 43s, weight download 3m 56s (31 GB plus the 3.8 GB drafter), and 5m 07s to load, compile and size the KV cache. A compile cache is machine code. The cache key has to include the GPU and the library versions, and only trusted machines should publish into a shared cache.

What was pinned

213 once is a demo. The recipe pins the FP8 weights to a commit, the drafter to a revision, the draft length, and vLLM 0.32.0 to an image digest. A combination counts only after it has come up healthy on that GPU type. That is the same rule the vLLM hosting page describes.

Caveats

One day, one laptop, 24 requests per provider. Speculative decoding is quicker on code than on open-ended prose. Your prompts will move the numbers. Precision differed: ModelRun was fp4, Parasail fp8, Cerebras fp16, Modal did not say. I ran FP8.

DFlash2’s own benchmarks, on an H200, one request at a time, report about 4.1 to 5.5 tokens accepted per step (GSM8K 5.46, HumanEval 4.39, MT-Bench 4.10), which they translate to roughly 2.7× to 3.4×. At 32 concurrent requests that shrinks toward 1.0× to 1.45×. Those are the drafter’s numbers, not another denv H100 run.