Longest verified prefix plus correction token.
Interactive inference systems laboratory
Draft fast. Verify once. Keep the exact distribution.
Speculative decoding lets a small model propose several tokens while the target model checks them in one parallel pass. Tune the proposal length and agreement rate, then watch accepted speedups and rejected work unfold in 3D.
One expensive pass can judge many guesses.
Ordinary autoregressive decoding calls the target model once per token. Speculation moves candidate generation to a cheaper draft model, batches verification, and uses a correction rule when the two distributions disagree. The output remains distributed like the target model; only the path to each token changes.
Ready: target context synchronizedDrag to orbit · wheel to zoom
draftedacceptedrejected
Sequential target calls replaced by one check.
Relative latency under the selected draft cost.
Proposals after the first rejection are discarded.
The draft model runs ahead
A cheaper model samples γ candidate tokens autoregressively. This work is useful only if the target considers those candidates plausible.
x₁…xγ ~ q(xᵢ | prefix, x<i)The target scores the whole branch
Because all proposed tokens are known, the target model evaluates their conditional probabilities in one parallel forward pass rather than γ sequential passes.
accept xᵢ with probability min(1, p(xᵢ) / q(xᵢ))The first rejection repairs the path
If a candidate is rejected, later draft tokens depend on a prefix that no longer exists and are discarded. A correction token is sampled from the positive residual between target and draft distributions.
x correction ~ normalize(max(0, p(x) − q(x)))The accepted prefix becomes context
When every candidate is accepted, the target can additionally sample one bonus token from its final position. High acceptance amortizes target latency over many emitted tokens.
Short drafts
Low wasted work and reliable progress, but fewer target calls are amortized per round.
Best when draft quality is uncertain.Long drafts
Large upside when models agree. One early rejection invalidates every proposal after it.
Best when the draft closely matches the target.Adaptive drafts
Confidence thresholds or learned policies stop drafting when acceptance becomes unlikely.
Best when agreement varies by context.