Speculative Decoding for Production LLM Serving

Speculative decoding is easy to describe and easy to misuse. A cheap drafter proposes several tokens, a stronger target model verifies them in one pass, and the serving stack accepts the matching prefix. That sounds like a free latency win until you remember that every extra moving part has to pay for itself in acceptance rate, batch shape, and scheduler behavior.

The cleanest way to think about it is this: you are not changing the final answer distribution in search of magic. You are trying to move work from an expensive model to a cheaper one without breaking correctness, then keep the target model busy enough that the extra draft step actually reduces wall-clock time. The original speculative sampling paper reports a 2-2.5x decoding speedup in a distributed setup, but TensorRT-LLM warns that the gains are only observable at low batch sizes and that there is no runtime switch to turn speculation off once enabled. Those two facts define the real game.

What the request path actually does

The diagram above is original. It shows the request path I would actually put in production: draft first, verify second, accept the prefix that matches, and only then stream the result. The point is not to make the loop look clever. The point is to make the expensive forward pass happen less often when the request shape allows it.

The loop is useful only when the drafter is cheap enough and accurate enough to reduce the number of sequential target passes. If the drafter guesses badly, you have added overhead without removing enough target work. If the batch is already large, the server may spend more time coordinating the speculation path than it saves on generation.

What the sources actually say

This screenshot matters because it anchors the only hard number I treat as a starting point here: the paper reports a 2-2.5x decoding speedup in a distributed setup. That is evidence for the idea, not a guarantee for your traffic mix.

This chart is original and based solely on the range reported in the paper abstract. It is intentionally boring. A range is more honest than a single heroic number.

The implementation docs fill in the operational details that matter after the paper stage. vLLM describes speculative decoding as a low-batch-size technique: a lightweight drafting mechanism proposes candidate tokens, the target model verifies them in a single forward pass, and accepted tokens reduce the number of sequential passes. The same docs also note that there is currently no way to dynamically disable speculation after it is enabled, so the speedup only shows up when the request mix and batch size are on your side.

TensorRT-LLM says the same thing in a more direct way: speculative decoding is for low batch sizes, and if the draft and target models do not share a tokenizer, acceptance rate collapses and performance regresses. That is the real line between a win and a lab demo.

Which method should you start with?

vLLM groups the available methods into two rough families. Model-based methods such as draft models, EAGLE, MTP, PARD, and MLP aim for the biggest latency cuts. Simpler methods such as n-gram and suffix decoding usually cost less to operate, but the gain is smaller.

My default rule is simple: start with the least clever method that still has a chance of moving p95 latency. If your workload is repetitive, n-gram or suffix decoding may be enough. If the traffic is broad and the model family supports it, draft-model speculation is the more general path.

A safe vLLM baseline

This is the shape I would start with for a general chat route. It keeps the config small, makes the tokenizer assumption explicit, and leaves room to measure acceptance rate before optimizing further.

1from vllm import LLM, SamplingParams
2
3llm = LLM(
4    "Qwen/Qwen3-8B",
5    speculative_config={
6        "method": "draft_model",
7        "model": "HuggingFaceTB/SmolLM2-135M-Instruct",
8        "num_speculative_tokens": 4,
9        "rejection_sample_method": "strict",
10    },
11    gpu_memory_utilization=0.5,
12)
13
14params = SamplingParams(
15    temperature=0.0,
16    max_tokens=256,
17)

The important part is not the exact model names. It is the shape of the contract. Keep the draft model small, keep the target model authoritative, and make sure the tokenizer story is clean before you chase throughput. If you need heterogeneous vocabularies, vLLM does expose a special path for that, but that is not where I would start.

For comparison, TensorRT-LLM exposes a draft-target config and also a native EAGLE 3 path. It explicitly says that matching tokenizer behavior matters and that speedups are only visible at low batch sizes. That combination makes speculative decoding a serving problem, not just a model problem.

1from tensorrt_llm.llmapi import DraftTargetDecodingConfig, LLM
2
3speculative_config = DraftTargetDecodingConfig(
4    max_draft_len=3,
5    speculative_model="yuhuili/EAGLE3-LLaMA3.1-Instruct-8B",
6)
7
8llm = LLM(
9    "/path/to/target_model",
10    speculative_config=speculative_config,
11    disable_overlap_scheduler=True,
12)

Where it pays off

The best candidates are requests where the target model is still doing the same expensive thing over and over: short chat turns, classification, extraction, intent routing, and template-heavy generation. Those are exactly the routes where a draft pass can replace several target passes without changing the user-visible output.

That also means speculative decoding pairs well with a few other patterns already on this site. If your prompts repeat a lot of static context, pair this with Prompt Caching in Real-World LLM Apps. If your server already depends on efficient KV management and batch expansion, read vLLM PagedAttention and Continuous Batching. If your output must be machine-readable, combine it with Structured Outputs: Reliable AI APIs. And if you want to know whether latency improvements are visible in production traces, use OpenTelemetry GenAI Observability: A Production Guide.

Where it does not pay off

Speculative decoding is a bad fit when the request is already dominated by batching, when the output is tiny, or when the acceptance rate is low enough that the draft model becomes dead weight. Tool calls are a common trap. If the route produces a few tokens of structured JSON and then hands control to a function, the draft work may not be enough to matter.

It is also a bad fit when you cannot keep the request path stable. If the scheduler changes batch shape constantly, if the tokenizer story is inconsistent, or if you are already near the memory edge, the added path complexity can erase the benefit. That is why the first question should be "what does acceptance rate look like on my route?" and not "how clever can I make the speculation tree?"

A routing rule I would actually ship

1type RequestShape = {
2  batchSize: number;
3  route: 'chat' | 'tool' | 'extract' | 'stream';
4  usesStructuredOutput: boolean;
5};
6
7export function shouldEnableSpeculation(req: RequestShape): boolean {
8  return req.batchSize <= 4 && req.route === 'chat' && !req.usesStructuredOutput;
9}

That rule is intentionally conservative. It assumes the interesting case is a user-facing chat route where the server still spends a lot of time in sequential decoding. If the result is machine-only, tiny, or heavily structured, I would measure first and turn it on later only if the data supports it.

Production checklist

• Measure acceptance rate per route, not just overall throughput.
• Track batch size alongside latency. If the batch grows, the benefit may shrink.
• Keep draft and target tokenizers aligned unless you have a reason not to.
• Start with one low-risk route, not the whole API surface.
• Log p50, p95, and retry count before and after the rollout.
• Put a rollback flag in front of the speculation path, even if the engine does not support dynamic disable inside the request.
• Validate output quality with the same checks you already trust for structured output or tool use.

FAQs

Is speculative decoding lossless?

In the theory sense, the target distribution is preserved by the rejection-sampling step. vLLM says speculative decoding is meant to improve inference efficiency while maintaining accuracy, but it also warns that floating-point precision and batch-size changes can still affect exact outputs. So the right answer is "lossless in the algorithmic sense, not magical in the real machine sense."

Why does low batch size matter so much?

Because the technique exists to remove sequential target passes, and batching already amortizes some of that cost. TensorRT-LLM is explicit that the speedup is only observable at low batch sizes. Once the server is already busy packing requests, the draft path may stop paying for itself.

Do the draft and target models need the same tokenizer?

In the simple draft-target setup, yes, that is the safe assumption. TensorRT-LLM says acceptance rate falls sharply if the tokenizers differ. vLLM does support a heterogeneous-vocabulary path, but that is a special case, not the baseline I would trust first in production.

Is a bigger draft model always better?

No. A better draft model can raise acceptance, but it also costs more to run. If the drafter is too heavy, you are just moving work around. The useful question is not whether the draft model is stronger; it is whether the combined draft-plus-verify path is faster than the target-only path on your route.

Should I start with n-gram or draft-model speculation?

Start with the simplest path that matches the traffic shape. If your requests repeat a lot of text, n-gram or suffix decoding is often enough. If the traffic is more general, draft-model speculation is the more flexible first test. The point is to buy back sequential passes, not to prove you can tune every knob.

Does this help with tool-calling or JSON output?

Sometimes, but only after measurement. If the route is short and already constrained by schema validation, the draft overhead may not buy enough. For routes that must emit strict machine-readable output, pair speculation with the same validation layer you already trust for structured outputs.

What should I watch in observability?

Look at latency, acceptance rate, retry rate, and batch size together. If p95 drops but retries climb, the path may be hiding quality loss. If acceptance stays low, the draft model is probably too weak or the request shape is wrong. The instrumentation matters as much as the model choice.

Bottom line

Speculative decoding is not a feature to flip on because the paper number looks nice. It is a serving strategy that only works when the request shape, tokenizer setup, and scheduler behavior line up. The paper gives you the hypothesis. The docs give you the operating constraints. Your traffic gives you the verdict.

If I were rolling this out tomorrow, I would start with one low-batch chat route, a small draft model, strict verification, and a rollback flag. If the chart moves and the traces stay clean, keep going. If not, leave the trick in the lab and spend the effort on caching, batching, or output shaping instead.