Docs · how to

Optimize on your traffic

Let denv search for a faster recipe on your real traffic, verify it against the official model, and ship the winner through your stages.

Published benchmark results are a good start, not the best your GPUs can do on your traffic. They use standard workload shapes and public GPU prices; your speed, throughput and cost depend on your own prompt lengths, how much of each prompt repeats, how many requests arrive at once, and the latency you need. The optimizer measures that traffic and searches for a recipe that serves it faster, then ships only what it has verified.

What it searches

Lever Examples
Engine settings Batch limits, scheduler settings, CUDA or HIP graph capture
Parallelism Tensor, data and expert parallelism; two smaller replicas instead of one large one
Topology Aggregated vs disaggregated; prefill and decode pool sizes
Decoding Speculative decoding on or off, draft length
KV cache Tiers, sharing, eviction (Configure the KV cache)
Quantization Formats the recipe allows. Off by default; only kept if parity passes.
Engine Another engine version, or another engine with a recipe for the model

1. Start

denv optimize start ds-flash/prod-us --goal throughput --budget-gpus 8

--goal is what to improve while staying inside the stage's alarms: throughput (tokens per second per GPU), latency (time to first token and inter-token latency), or cost (cost per million tokens). --budget-gpus caps the spare GPUs the optimizer may use for trials. It never takes GPUs serving customers.

2. Watch candidates

denv optimize candidates ds-flash/prod-us

CANDIDATE   CHANGE                              TOK/S/GPU   TTFT P99   PARITY     STATUS
cand_01     spec decoding on                    +          =          512/512    verified
cand_02     2× TP4 instead of 1× TP8            +          -          512/512    verified
cand_03     fp8 kv cache                        +          =          497/512    rejected

Each candidate is replayed on recorded traffic shapes (lengths and timing, not prompt contents), then run on a canary against the parity suite. A candidate that changes answers is rejected, however fast it is.

3. Ship the winner

denv optimize promote cand_01

This creates a release from the candidate and deploys it to the environment's first stage. From there it moves through your pipeline like any other release: bake, alarms, approvals, automatic rollback.

To ship verified winners without a command, start with --auto. Promotion rules on each stage still apply.

4. Keep it running

Traffic changes, new engine versions ship, and new models arrive. Leave the optimizer running and it measures again when any of these change.

denv optimize stop ds-flash/prod-us

What it never does

  • Use GPUs serving customer traffic for trials.
  • Ship a candidate that failed parity, or skip your stages and approvals.
  • Store prompt contents. It records lengths, timing and cache reuse.