The profiler said the kernel ran zero times

The 23% kernel from the previous note did not make Qwen3.8-27B faster. I assumed it was running. A profiler trace said it ran zero times.

Adapted from the article on X. The long version, with the kernel and the tests, is the other article. This post is the finding, not the CUDA.

What the trace showed

I recorded about 60 steps with PyTorch’s profiler. Two facts:

The matmuls were not on CUTLASS, which is what I had benchmarked against. vLLM was sending them to DeepGEMM, which is already built for small batches. My 23% was a win over a kernel that was not in the race.

And my kernel’s name was absent.

Roughly where one step went:

Where	Share
FP8 matmuls, DeepGEMM	about 57%
Output layer, still 16-bit	about 15%
Linear attention	6.5%
GPU idle, waiting on the CPU	about 4%
Full attention	3.5%
Casting inputs to FP8	2.8%
Everything else	about 11%

Full attention was 3.5% on this trace and about 19% on long agent sessions, because it rereads the whole history. The biggest DeepGEMM layer was already at about 78% of the H100’s peak memory bandwidth. A perfect kernel there does not change the model much.

Why it never ran

I got the first theory wrong. I launch these engines from a recipe. The setting that enables the plugin did not show up when I fetched the recipe back, so I thought the launcher had dropped it. That view never shows the setting, for any recipe. A missing field in a view is not proof the field is gone.

The test now looks inside the running process before it trusts a number. Is the setting on? Can vLLM see the plugin? Did the plugin load? If any answer is no, the run stops.

That check found it. vLLM reads a plugin’s Python type hints to learn the inputs and outputs. The file started with from __future__ import annotations, so those hints were stored as strings. vLLM could not turn the strings back into types, and registration failed. The plugin caught the error, logged a warning, and let the engine start. I wrote it that way so a broken kernel could not take down an engine. That is why nobody noticed.

Even if it had registered, it was hooked to the CUTLASS path. These layers do not take that path.

The output layer, which did run

The output layer turns the model’s last state into a score for every token in the vocabulary. This model’s vocabulary is 248,320, and the layer is a 248,320 by 5,120 matrix. FP8 checkpoints usually leave it in 16-bit, because those scores pick the token. That was about 2.5 GB read every step, and about 15% of the GPU’s time.

I converted it to FP8 the same way as the rest, one scale per 128 by 128 block. On the real weights it went from 1.78 ms to 0.88 ms per call, 2.0×. In the profile, that layer’s share went from 10.4% to 5.7%. The draft model’s output layer was still 16-bit, about another 7%. I left it.

End to end, one request at a time, same GPU and settings:

	Before	After
Chat	442 tok/s	468 (5.7% more steps)
Edit	398 tok/s	440 (7.3% more steps)
16 requests at once	baseline throughput	about 3% higher

Chat is the clean number. Tokens accepted per step stayed 6.34 then 6.35, so the gain was faster steps. On an agent workload, tokens per step also jumped from 4.46 to 5.15, and I cannot explain that, so I am not counting it.

The catch

FP8 rounding can change the text. Thirty prompts, greedy decoding, so the model always picks its top token. Thirteen answers matched. Seventeen drifted: “This mechanism allows” became “enabling”, “How it works” became “Explanation”. The facts I checked were the same. Two tokens with almost the same score can flip, and the sentence goes somewhere else.

That is a real trade. I will not put it on live traffic until an evaluation says the answers are as good, not merely similar on thirty prompts. The 213 tok/s path did not need this. It kept the FP8 checkpoint’s answers and changed the drafter.

Profile before you write the kernel. Benchmark the library that is actually running. Check the process, not a view of the recipe.