Parity

The same open weights can be served in many ways: by you on different engines and GPUs, or by many providers. The model name does not tell you whether a deployment serves them faithfully. Parity is how you check, and how you show others. Parity is a deployment gate; it is not part of benchmarks, which measure speed, throughput and cost.

Why outputs differ between providers

Serving the same weights can still produce different results:

• Quantization. Lower precision weights or KV cache change outputs, sometimes a little, sometimes a lot.
• Chat templates. A template that differs from the official one changes how the model sees the conversation.
• Tool-call and reasoning parsing. The model's raw output has to be parsed into tool calls, reasoning and content. Parsers differ.
• Sampling and limits. Default max_tokens, stop sequences, and how temperature and top_p are applied.
• Engine bugs and kernels. Numerical differences between engines and versions.

For DeepSeek models, DeepSeek released deepseek-recipe so community deployments can match the official API's request and response handling. denv uses it as the API adapter for those models. Parity checks that the result actually matches.

What a parity run compares

A parity suite is a JSONL file, one case per line. Each case has an id, Chat Completions request fields (messages, and optionally tools and response_format), and an expect. Cases run at temperature 0. denv sends each case to your stage and to the reference, then checks:

{"id": "math-1", "messages": [{"role": "user", "content": "What is 17 * 23?"}], "expect": {"contains": "391"}}
{"id": "weather-tool", "messages": [{"role": "user", "content": "Weather in Paris?"}], "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}], "expect": {"tool-call": {"name": "get_weather", "arguments": {"city": "Paris"}}}}
{"id": "greeting-lp", "messages": [{"role": "user", "content": "Say hello."}], "expect": {"logprobs-close": {"tolerance": 0.05}}}

expect	Passes when
equals	Output text equals the string.
contains	Output text contains the string.
regex	Output text matches the pattern.
tool-call	The response calls the named tool with these arguments. Extra arguments are allowed.
logprobs-close	Token log-probabilities match the reference's within tolerance, where both sides return them.

For equals, contains and regex, the reference's answer must pass too; a case the reference fails is reported, not counted against you.

The reference is the model's official API or a deployment you trust, set in the release's parity.reference. The suite comes from the recipe or your own file.

Deterministic sampling

Token-level comparison only works if the same input produces the same tokens. On most engines, results can change with batch composition. Engines that offer batch-invariant inference and deterministic sampling, such as DeepInfer, make equals and logprobs-close checks meaningful even under load. On other engines, rely mostly on tool-call, contains and regex cases.

Thresholds

Set parity.pass on the release as the share of cases that must pass per kind, for example { tool-calls: 99%, exact: 95% }. tool-calls counts tool-call cases; exact counts equals, contains and regex cases. A run below any threshold fails.

When parity runs

• As a deployment gate, after each batch's health checks and golden prompts, before bake. A failure raises the parity-fail alarm and rolls back. A parity-fail during bake also rolls back, and the release cannot be promoted. See Rollout policy.
• On a schedule, with denv parity schedule, to catch drift in either side.
• On demand, with denv parity run.

See Run parity checks.

Reports

Each run produces a report: suite, reference, release, engine and recipe versions, pass rates per check, and failing cases. denv parity report --public gives a URL you can share with customers or list alongside your endpoint. A public report contains only your suite's cases, never prompts from customer traffic.

Limits

Parity is evidence, not proof:

• It covers the suite's prompts, not every prompt.
• The reference can change or be wrong.
• On engines without deterministic sampling, text checks can fail by chance and need tolerant thresholds.
• A provider can serve differently outside parity runs. denv's router sends parity requests through the same path as customer traffic, so a stage cannot special-case them, but buyers must still trust whoever runs denv.

Use it to catch regressions early and to give buyers something better than a model name.