Configure the KV cache

The KV cache holds the attention state of tokens already processed. When a new request starts with tokens the cache already holds, such as a system prompt, tool definitions or the earlier turns of an agent conversation, that part of prefill is skipped. Two settings decide how much is reused: where the cache lives (tiers) and who can reuse it (sharing).

Tiers

release:
  kv-cache:
    tiers: [gpu, host-memory, nvme]
    share: stage

Tier	Capacity	Reuse cost	Notes
gpu	Smallest	None	Competes with batch size for GPU memory
host-memory	Larger	Copy to GPU	Default second tier
nvme	Largest	Read from disk, then copy	Needs local NVMe on hosts; denv host list shows it

Blocks move down the tiers as they age and up again when a request needs them. The default is [gpu, host-memory]. Add nvme when your workload has many long prefixes that come back after minutes rather than seconds.

Sharing

share	Behavior
replica	Each replica reuses only its own cache.
stage (default)	Prefixes are found across all replicas of the stage, and the router sends requests where their prefix already is.

With stage, the router looks up how much of each request's prefix every replica holds, then balances that against queue depth and the stage's routing policy. A replica that holds the prefix but is overloaded does not get the request.

Agent workloads

Agents send long, growing conversations: the same system prompt and tools, plus every earlier turn. Most of each request is a prefix of the last one. To make that reuse work:

• Keep share: stage so the next turn can land anywhere and still hit.
• Add host-memory, and nvme if conversations pause for minutes between turns.
• Ask customers to keep stable content (system prompt, tools) at the start of the request and put variable content at the end.

Cached input pricing

If you sell tokens, you can price cached input lower than fresh input, because it costs you less:

denv price set --env ds-flash --input 0.30 --cached-input 0.03 --output 1.10

These numbers are placeholders; set your own. Usage reports cached-input tokens separately. See Sell tokens.

Measure

denv metrics ds-flash/prod-us

Look at:

Metric	What it tells you
KV cache hit rate	Share of input tokens served from cache
Hit rate by tier	Whether lower tiers earn their place
Time to first token	Should fall as the hit rate rises
Evictions	Cache too small for the working set

Sizing trade-offs

• More GPU memory for cache means less for batching. If throughput drops after a cache change, the GPU tier is taking too much.
• Lower tiers add capacity at the cost of copy time. They pay off only if hits there are common.
• Disaggregated releases move KV from prefill to decode anyway; sharing at stage level lets a decode replica skip prefill entirely for a cached prefix. See Disaggregate prefill and decode.

Cache settings are part of the release, so changing them is a new release with a canary. See Serving topology.