Prove your endpoint serves the model it claims to, by comparing it with a reference API on the same prompts, as a deployment gate and on a schedule.

Many providers serve the same open weights. Callers cannot see whether a provider quantized the model, used the wrong chat template, or runs an engine version with a bug. A parity suite sends the same prompts to your endpoint and to a reference, and compares the answers.

1. Store the reference key

denv secret set deepseek-api-key --value <key>

The reference is usually the model's official API. It can also be another deployment you trust, such as a reference release on your own fleet.

2. Write a suite

A suite is a JSONL file, one case per line. Each case has the request fields of a Chat Completions call plus expect.

{"id": "capital", "messages": [{"role": "user", "content": "Capital of France? One word."}], "expect": {"equals": "Paris"}}
{"id": "json", "messages": [{"role": "user", "content": "Return {\"ok\": true}"}], "response_format": {"type": "json_object"}, "expect": {"regex": "^\\{\\s*\"ok\"\\s*:\\s*true\\s*\\}$"}}
{"id": "weather-tool", "messages": [{"role": "user", "content": "Weather in Oslo?"}], "tools": [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}}}], "expect": {"tool-call": {"name": "get_weather", "arguments": {"city": "Oslo"}}}}
{"id": "logprobs", "messages": [{"role": "user", "content": "Count to five."}], "expect": {"logprobs-close": {"tolerance": 0.05}}}

`expect`	Passes when
`equals`	Both outputs equal this string after trimming
`contains`	Both outputs contain it
`regex`	Both outputs match it
`tool-call`	Both call this tool with these arguments (extra arguments allowed)
`logprobs-close`	Your token log-probabilities are within `tolerance` of the reference's for the reference's tokens

Cases run at temperature 0. Cover what your applications depend on: tool calls, structured output, reasoning on and off, long context.

3. Use deterministic sampling where you can

logprobs-close is the strictest check. It needs the reference to return log-probabilities, and works best when your engine is deterministic. DeepInfer's README lists batch-invariant inference and deterministic sampling among its features; Engines shows which engine versions support them. Without determinism, use equals, contains, regex and tool-call.

4. Configure it on the release

release:
  parity:
    reference: { url: https://api.deepseek.com, secret: deepseek-api-key, model: deepseek-v4.1-flash }
    suite: parity/flash.jsonl
    pass: { tool-calls: 99%, exact: 95% }

pass sets the share of cases that must pass per kind. A deployment runs the suite after health checks and golden prompts, before bake. Below the thresholds, the deployment stops and rolls back. Add the alarm so a scheduled failure also counts:

alarms: { parity-fail: 1 }

5. Run it now, and on a schedule

denv parity run ds-flash/prod-us

suite        parity/flash.jsonl  412 cases
equals       188/190
tool-call    120/120
regex        51/52
logprobs     50/50  (tolerance 0.05)
result       pass

denv parity schedule ds-flash/prod-us --every 6h

denv parity report ds-flash/prod-us --public

This prints a URL showing the suite, the reference, the release and the results, with no prompts from customer traffic. Give it to customers who ask whether you serve the real model. See Sell tokens.

What failures usually mean

Pattern	Likely cause
Tool calls missing or malformed, plain text fine	Wrong adapter or chat template. Check the recipe's `adapter`.
Small drift everywhere, `logprobs-close` failing	Quantized weights or a different precision than the reference
One kind of case fails after an engine upgrade	Engine regression; roll back and pin the previous release
Fails on one fleet only	That fleet's recipe or hardware differs; compare with `denv recipe get`
Reference changed, you did not	The reference updated its model; check its release notes before chasing your own

See Parity for how comparison works.

Run parity checks