vLLM PagedAttention and Continuous Batching

Methodology

This guide uses primary sources only: the PagedAttention paper, vLLM documentation, and NVIDIA TensorRT-LLM documentation. Any throughput number below is paper-reported or vendor-reported. I did not run a local benchmark for this article, so every claim should be read as workload-dependent rather than universal truth.

Why KV cache is the real bottleneck

Serving an LLM looks like a compute problem from a distance, but in production it often behaves like a memory-management problem with a neural network attached.

During decoding, each active sequence needs key/value tensors for every layer so the model can keep attending to the prior context. That KV cache grows with prompt length and with the number of concurrent requests. If the serving stack reserves memory naively, three things happen fast:

1. Head-of-line blocking appears when short requests wait behind long ones.
2. Fragmentation grows because sequences finish at different times and leave unusable holes in memory.
3. Effective batch size shrinks because you cannot safely fit as many live sequences as the raw GPU memory budget suggests.

That is why “faster inference” is not just kernel tuning. The scheduler, memory allocator, and decode strategy determine how much useful work each GPU step actually does. A good serving system keeps the GPU busy without forcing every request into the same rigid shape.

What PagedAttention changes

The PagedAttention paper attacks the cache problem directly. Instead of treating each sequence as one contiguous memory reservation, it splits KV cache into fixed-size blocks and maps them through a block table, much like virtual memory pages in an operating system.

That gives you two practical wins:

1. You allocate cache on demand instead of reserving a giant contiguous slab up front.
2. You can reuse and share blocks more flexibly when prompts share prefixes or when multiple requests need the same cached history.

This screenshot proves the paper’s own abstract-level claim: PagedAttention is about reducing KV cache waste, and the authors report a 2-4x throughput improvement at the same latency compared with FasterTransformer and Orca. That is a paper-reported result, not an independent reproduction.

Fixed blocks instead of contiguous reservations

The block-based design matters because real traffic is messy. A chat request might be 40 tokens long; the next one might be 4,000 tokens. With contiguous reservations, you tend to over-allocate for the worst case and leave the GPU underutilized. With block management, unused tail space does not poison the whole cache.

This is also where the operating-system analogy stops being a metaphor and starts being useful. The point is not that the GPU literally runs a desktop-style page table. The point is that indirection lets the runtime decouple logical sequence length from physical memory layout.

Prefix caching is not a bonus feature

Once you have block-based cache management, repeated prefixes become valuable. Many production systems reuse the same system prompt, policy prompt, retrieval scaffold, or agent instructions across many requests. vLLM’s automatic prefix caching reuses KV-cache blocks for shared prefixes so the model does not recompute the same context over and over.

This screenshot proves the implementation detail that matters operationally: prefix caching is documented as KV-cache block reuse for shared prefixes, not just “smarter prompting.” If your workload repeats system prompts, agent instructions, or tool-routing prefixes, the cache hit rate can be material.

Continuous batching keeps the GPU from waiting

The second half of the story is scheduling. Continuous batching, also called iteration-level scheduling, means the runtime does not freeze a batch until every sequence is done. It reevaluates the roster at each decode step and admits new work as soon as a slot opens.

That is a much better fit for real traffic than static batching. Static batching is simple, but it waits for the slowest sequence in the batch. Continuous batching lets short requests finish, frees their cache blocks, and backfills the next request without stopping the machine.

In practice, this matters most when you have:

1. Mixed prompt lengths.
2. Streaming chat responses.
3. Agent loops with many short intermediate calls.
4. A queue of small requests arriving steadily instead of one giant offline job.

Why this is different from ordinary batching

Ordinary batching is frame-based: you collect a set of requests, run them, then collect the next set. Continuous batching is step-based: the batch can change after every decode iteration.

That sounds like a subtle implementation detail. It is not. It changes the shape of the latency curve:

• Short requests stop paying for long tail requests.
• GPU utilization stays high because freed slots are reused quickly.
• Queueing delay becomes a scheduler problem instead of a hard wall.

If you want a mental model, think of static batching as a train that leaves only when every passenger boards. Continuous batching is a metro system: every time a car empties, the next rider gets on.

Speculative decoding is a separate lever

PagedAttention and continuous batching improve how the serving stack uses memory and scheduling. Speculative decoding tackles something else: the number of expensive target-model passes required to produce output.

The idea is simple. A smaller draft model proposes several tokens ahead. The larger target model verifies them. If enough draft tokens are accepted, you effectively amortize expensive decode work over multiple output tokens.

vLLM documents several speculative-decoding methods and notes that the feature is not universally compatible with every parallelism setup. That is the part operators should remember. The right question is not “does speculative decoding exist?” The right question is “does it work for this model, this topology, and this latency target?”

When speculative decoding helps

It helps most when:

1. The draft model is cheap and accurate enough to predict useful tokens.
2. The target model is expensive relative to the draft model.
3. You care about interactive latency, not just aggregate throughput.

When it hurts

It can hurt when:

1. The draft model is too weak and most proposals get rejected.
2. The extra coordination cost outweighs the saved target passes.
3. The serving topology or parallelism mode disables the best path.

The lesson is the same one every production AI system eventually learns: the feature is only a win if the whole pipeline is measured, not just the model.

The architecture in one pass

This diagram shows the production path I would actually reason about: request routing, step-wise scheduling, block-based cache management, optional prefix reuse, and a clean streaming response boundary. It is not a benchmark. It is the decision tree you need before you tune anything.

What the paper-reported gain actually means

The PagedAttention paper’s abstract reports a 2-4x throughput improvement at the same latency compared with FasterTransformer and Orca. That is a useful headline, but it is not a license to copy the number into your own architecture review.

The chart intentionally stays conservative. It shows the published range instead of pretending to know your exact gain. That is the right editorial standard for AI infrastructure claims: use the source’s own numbers, keep the uncertainty visible, and test your workload separately.

vLLM versus TensorRT-LLM

vLLM and TensorRT-LLM solve a similar problem from different angles.

The NVIDIA docs are explicit that TensorRT-LLM includes in-flight batching, paged attention, speculative decoding, and paged KV-cache management. That puts it in the same family of ideas as vLLM, but the deployment story is different. If your environment is already optimized around NVIDIA GPUs and TensorRT workflows, it belongs on the shortlist. If you want a fast evaluation path and a broad open-source serving surface, vLLM is often the easier place to start.

This screenshot proves the comparison point directly from NVIDIA’s own documentation: the runtime is built around in-flight batching, paged attention, paged KV cache management, and speculative decoding. That makes it a relevant alternative, not an abstract competitor.

A practical deployment pattern

If I were rolling this out in production, I would start with a controlled A/B test, not a wholesale swap.

1vllm serve meta-llama/Llama-3.1-8B-Instruct \
2  --enable-prefix-caching \
3  --max-num-seqs 128 \
4  --max-num-batched-tokens 8192

The exact flags will vary by version and model, but the point is stable: enable the cache optimizations you intend to test, then hold the rest of the stack constant.

1import asyncio
2import time
3from openai import AsyncOpenAI
4
5client = AsyncOpenAI(
6    base_url="http://localhost:8000/v1",
7    api_key="EMPTY",
8)
9
10async def one(prompt: str):
11    start = time.perf_counter()
12    response = await client.chat.completions.create(
13        model="meta-llama/Llama-3.1-8B-Instruct",
14        messages=[{"role": "user", "content": prompt}],
15        temperature=0,
16        max_tokens=128,
17    )
18    elapsed = time.perf_counter() - start
19    return elapsed, response.choices[0].message.content
20
21async def main():
22    prompts = [
23        "Summarize this incident in three bullets.",
24        "Classify the ticket as billing, auth, or infrastructure.",
25        "Rewrite this query for better retrieval.",
26    ]
27    results = await asyncio.gather(*(one(p) for p in prompts))
28    for latency, text in results:
29        print(f"{latency:.3f}s :: {text[:80]}")
30
31asyncio.run(main())

That test does two useful things. First, it exercises concurrency so you can see whether continuous batching actually helps under load. Second, it gives you a fixed evaluation harness for latency, output shape, and quality. If you care about schema adherence, pair the same harness with the stricter contract patterns from my Structured Outputs guide. If the model is allowed to call tools, keep the tool boundary as tight as the one described in my MCP threat modeling guide.

What to measure before you trust the win

Do not ship on vendor adjectives. Measure:

• p50, p95, and p99 end-to-end latency.
• Time to first token for streaming workloads.
• Cache hit rate for repeated prefixes.
• First-pass structured-output success if you return JSON.
• Queue depth and cancellation rate at peak concurrency.
• GPU memory headroom after warmup.

That telemetry also belongs in your traces. My OpenTelemetry GenAI observability guide covers the discipline I would reuse here: model choice, prompt version, timing, fallback path, and final outcome should all be observable.

Common failure modes

There are a few ways this kind of stack can look good in a demo and disappoint in production:

1. The workload has low prefix reuse, so prefix caching barely helps.
2. The model is too large for the memory budget, so batching gains vanish under pressure.
3. The draft model for speculative decoding is not good enough to earn back its overhead.
4. The comparison uses mismatched prompts, temperatures, or hardware, so the numbers are not apples-to-apples.
5. The team measures model latency but ignores queueing delay, which is where batching behavior really shows up.

If the target workload is agentic, the surrounding system matters just as much as the model. The scheduling lessons in my AI Agent Stacks article apply here: keep the fast path narrow, make the expensive path deliberate, and never confuse a clever model with a complete system.

FAQ

Is PagedAttention the same thing as FlashAttention?

No. FlashAttention is primarily about making attention computation more efficient. PagedAttention is about managing KV cache memory more efficiently during serving. They can live in the same stack, but they solve different bottlenecks.

Does continuous batching make one request slower?

Not automatically. It can add scheduler complexity, but the point is to reduce overall queueing and keep the GPU busy. For mixed workloads, the end-to-end result is often better than waiting for a fixed batch to drain.

When does prefix caching matter most?

It matters most when many requests share the same system prompt, policy prompt, tool scaffold, or retrieval preamble. If every request is unique, the hit rate will be low and the benefit will shrink.

Is speculative decoding always worth it?

No. It is a trade between draft-model cost, acceptance rate, and the overhead of verifying proposed tokens. It is powerful when the draft model is cheap and accurate enough, but it should be measured on your exact workload.

Should I choose vLLM or TensorRT-LLM?

Pick vLLM when you want an accessible open-source serving path, fast experimentation, and strong cache-oriented optimizations. Pick TensorRT-LLM when your deployment is deeply centered on NVIDIA GPUs and you want the runtime and quantization features NVIDIA exposes directly.

What should I benchmark first?

Benchmark the traffic shape that actually hurts you: repeated agent prompts, long-context chat, short high-concurrency requests, or mixed prompt lengths. If the workload is mostly homogeneous offline generation, continuous batching may matter less than kernel throughput.

What is the biggest production mistake teams make here?

They benchmark a single happy-path prompt and treat it as a deployment result. For serving systems, the shape of the queue matters as much as the model answer.

Sources and related reading

Primary sources used for this article:

• PagedAttention paper on arXiv
• vLLM automatic prefix caching docs
• vLLM speculative decoding docs
• TensorRT-LLM overview

Related internal reads:

• OpenTelemetry GenAI Observability: A Production Guide
• Structured Outputs for Reliable AI APIs
• AI Agent Stacks: skills, plugins, MCP, ACP, memory, and workflows
• MCP Security Threat Modeling Guide