A 23% faster GPU kernel, and a model that did not move

I wrote an FP8 matrix-multiply kernel for Qwen3.8-27B on one H100. In isolation it was 23% faster than the kernel I thought vLLM was using. In the actual model, tokens per second did not move.

This is the short version of that night, adapted from the X article. The part where a profiler showed the kernel ran zero times is the next post. The code and the tests live in those articles. I am not pasting a kernel into the blog.

Why this shape is hard

Speculative decoding on this model was handing the GPU tiny batches, about 8 rows. vLLM’s FP8 kernel left about two thirds of the H100 idle on those shapes. The card was waiting on memory, and the work was too narrow to fill the 132 streaming multiprocessors.

The kernel I wrote splits the long dimension across those SMs, uses 128×128 block scales on the tensor cores, and reduces deterministically. That is the whole design, at the level that matters for the result.

The microbenchmark

Against the default FP8 path, at 8 rows:

Layer	Before	After
MLP up projection	81.2 µs	63.4 µs (1.28×)
MLP down projection	50.1 µs	35.7 µs (1.40×)
Attention output	20.0 µs	11.0 µs (1.82×)
Linear attention input	34.0 µs	33.1 µs (1.03×)
Attention QKV	31.4 µs	29.9 µs (1.05×)

Across the 256 multiplies in one forward pass, 11.8 ms down to 9.1 ms. 23% less. On these shapes the new kernel used about 73% to 87% of H100 memory bandwidth. The default used about 47% to 82%. At one row, without speculative decoding, the gain was only 4.5%. The default is already near the limit there.

The prediction, then the model

I wrote the end-to-end expectation down first. The engine was doing about 59 steps a second, about 17 ms a step, on the edit workload with 6.65 draft tokens accepted per step. Saving 2.7 ms of a 17 ms step should have been about 16% more tokens per second.

Same H100, same model, same settings, same draft model, measured on the machine so the network was not in the number:

Workload, one request	Without	With
Agent	354 tok/s	356
Edit	395	395
Chat	432	432

Sixteen requests at once moved a few percent, including a chat run that went slightly down (2,072 to 2,059). Flat, against a 16% prediction.

Amdahl’s law explains a small win. It does not explain zero. The likely reason, which I had not proved yet that night, was that the engine was not calling the kernel on the hot path. vLLM has more than one way to run an FP8 matmul. A plugin that loads is not a plugin that runs.

What I would repeat

Write the number you expect before you look at the result. If I had not written down 16%, I would have blamed Amdahl and stopped. A microbenchmark does not tell you the kernel is on the hot path. Compare on the same machine, in the same hour. My first baseline was from a laptop over the network and was not a baseline.

The speed that did show up that week was not this kernel. It was precision and a draft model: 50.8 to 213 tok/s. Kernels are allowed to lose.