Inside vLLM: How PagedAttention Changes Python LLM Inference at the Systems Level

You load a 7B model onto a 24 GB GPU, it fits fine, and then three concurrent requests arrive — and suddenly you’re watching OOM errors or a throughput cliff that makes no sense given how much memory you thought you had free. The model isn’t the problem. The allocator is.

If you’re doing Python LLM inference optimization with vLLM, PagedAttention is the mechanism worth understanding first — before you touch a single serving parameter. It treats GPU memory the same way an operating system treats RAM: demand-paged blocks allocated on arrival, not a contiguous reservation sized for the worst case. Once you understand how it works at the systems level, the config knobs stop feeling arbitrary.

By the end of this post, you’ll understand exactly what PagedAttention does to your GPU memory, which vLLM parameters control it, and how to tune a Python RAG pipeline to actually benefit from it.

---

The Real Reason Your GPU Memory Is “Full” When It Isn’t

Before vLLM, most inference frameworks pre-allocated a contiguous block of GPU memory for each incoming request’s KV cache — sized for the maximum possible sequence length, not the actual one. Your request might generate 200 tokens, but the allocator reserved space for 2,048.

Here’s what the KV cache actually costs. For each sequence, memory usage scales as:

2 × num_layers × seq_len × hidden_dim × dtype_bytes

For a 13B parameter model with 40 layers, float16, and a 2K context window, that’s roughly 2 GB per sequence just for the cache. Under concurrent load, the memory fills up fast — but not because it’s being used. It’s being held.

The Fragmentation Problem

Two kinds of fragmentation compound the issue:

• Internal fragmentation: a request that only needs 300 tokens still occupies the full 2,048-token reservation. That gap is permanently dead weight until the sequence finishes.
  
• External fragmentation: after several sequences complete, the freed blocks are scattered across GPU memory in non-contiguous chunks. A new request needing a large contiguous allocation can’t use them — even though total free memory looks adequate.
  

The vLLM research paper measured 60–70% GPU memory waste under these prior allocators. That’s not an edge case. That’s the baseline you’re working against every time you deploy a model without PagedAttention.

---

PagedAttention: What the OS Figured Out Decades Ago

Your operating system solved this exact problem for RAM in the 1960s. The insight was simple: processes don’t need physically contiguous memory — they need the illusion of contiguous memory. So the OS divides RAM into fixed-size pages, keeps a page table mapping virtual addresses to physical frames, and allocates pages on demand.

PagedAttention applies the same model to the KV cache — it’s a demand-paged block allocator, not a pre-allocator for full sequences.

Instead of one giant contiguous allocation per sequence, vLLM divides GPU memory into fixed-size blocks (16 tokens per block by default). Each sequence gets a block table — a lightweight mapping from logical block indices to physical GPU memory blocks. As tokens are generated, new blocks are appended to the table. Non-contiguous physical blocks appear contiguous from the attention kernel’s perspective.

Here’s a minimal view of what this looks like in practice:

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    gpu_memory_utilization=0.90,   # fraction of VRAM for the block pool
    max_model_len=4096,
    block_size=16,                 # tokens per KV block
    max_num_seqs=64,               # max concurrent sequences
    enable_prefix_caching=True,
)

No request pre-allocates 4,096 tokens of KV cache upfront. Blocks are claimed as generation proceeds and released immediately when a sequence finishes.

Copy-on-Write Makes Beam Search Cheap

Here’s where it gets interesting. When you run beam search or set n > 1 for parallel sampling, multiple output sequences branch from a shared prompt prefix. Under old allocators, each branch got its own full KV cache copy — expensive.

PagedAttention uses reference counting instead. Shared prefix blocks point to the same physical memory. Only when a branch writes a new token does it trigger a copy of that specific block. The rest stays shared. For a beam width of 4, you’re not paying 4× the memory for the shared portion of the sequence.

---

Python LLM Inference Optimization with vLLM: Parameters That Actually Matter

The LLMEngine runs a continuous batching loop internally. At each step, the Scheduler decides which sequences get GPU blocks. The BlockAllocator manages the pool — think of it as the OS page frame table. When memory pressure hits, vLLM can either swap a sequence’s blocks to CPU RAM or recompute them from scratch on the next step.

The swap-vs-recompute tradeoff matters most under bursty load. Swapping preserves the KV cache but costs PCIe bandwidth. Recomputation is cleaner but expensive for long sequences. vLLM defaults to recomputation; you generally don’t need to change this unless you’re hitting severe preemption rates.

The parameters worth knowing:

Parameter	Default	What It Controls
gpu_memory_utilization	0.90	Fraction of VRAM reserved for the block pool
block_size	16	Tokens per KV block (smaller = less internal fragmentation, more table overhead)
max_num_seqs	256	Max sequences in flight per scheduler step
max_model_len	model-dependent	Hard cap on sequence length; bounds per-sequence block count
enable_prefix_caching	False	Reuse blocks across requests with matching prefixes

One thing worth flagging: gpu_memory_utilization=0.90 doesn’t mean 90% of total GPU memory goes to the KV cache. It means 90% of the memory remaining after the model weights are loaded. On a 24 GB GPU with a model consuming 16 GB, you’re working with about 7.2 GB for the block pool. That’s easy to misread, and a lot of OOM debugging sessions start exactly there.

For Python local LLM development, tools like Ollama abstract all of this away — which is genuinely useful when you’re iterating on prompts and don’t need concurrent serving. But Ollama has no PagedAttention and no continuous batching. Once you move from a single dev machine to serving more than one user, vLLM’s approach pays off immediately.

---

PagedAttention in a Python RAG Application

Beyond local development, this is where the architecture pays off most visibly. If you’re building a Python RAG application using LlamaIndex, prefix caching is probably the highest-value parameter you’re not using yet.

RAG workloads have a consistent pattern: each request prepends a retrieved document chunk (or a few) before the user’s query. If multiple users query the same document, or your system uses a fixed system prompt, those prefix tokens are identical across requests. Without prefix caching, vLLM allocates fresh KV blocks for those tokens every single time and runs the full attention computation for them.

With enable_prefix_caching=True, vLLM hashes the content of each block. When a new request arrives with a matching prefix, the cached physical blocks are reused directly — no recomputation, no re-allocation.

from llama_index.llms.vllm import Vllm

llm = Vllm(
    model="meta-llama/Llama-3.1-8B-Instruct",
    vllm_kwargs={
        "enable_prefix_caching": True,
        "max_model_len": 8192,
        "gpu_memory_utilization": 0.88,
    },
)

The practical implication: standardize your system prompts and document chunking. If your RAG pipeline builds slightly different prefix strings for each query — timestamps, user IDs embedded in the prompt, dynamic preambles — you’re breaking the hash match and losing the cache benefit entirely. Consistency is the whole game here.

---

Tuning in Practice: Where to Start

Don’t adjust parameters blindly. Check the metrics first.

vLLM exposes a Prometheus-compatible endpoint at /metrics. The two numbers to watch:

• vllm:gpu_cache_usage_perc — if this stays below 50% under your real load, you have room to increase max_num_seqs
  
• vllm:num_preemptions_total — a rising count means the scheduler is running out of blocks and thrashing; lower max_num_seqs or max_model_len
  

Start here:

# Check block allocation at startup
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 64 \
  --enable-prefix-caching

# In another terminal, watch metrics
curl -s http://localhost:8000/metrics | grep vllm:gpu_cache

The startup log will also print num_gpu_blocks: N — that’s your total physical block budget. Divide by your typical sequence length in blocks to estimate your actual concurrency ceiling. It’s a rough number, but it’s a much better starting point than guessing.

---

Frequently Asked Questions

What is the difference between vLLM’s PagedAttention and standard KV cache management?

Standard KV cache implementations pre-allocate a single contiguous GPU memory region per sequence, sized for the maximum context length. vLLM’s PagedAttention splits that cache into fixed-size blocks allocated on demand and tracked through a block table — the same way an OS manages virtual memory. This eliminates both internal fragmentation (unused space within a reservation) and external fragmentation (scattered free space that can’t serve new allocations), resulting in significantly higher memory efficiency and throughput under concurrent load.

Should I use vLLM or Ollama for Python LLM development?

For local experimentation, Ollama is the better starting point — it’s straightforward to set up, handles model downloads cleanly, and works well for single-request workflows where latency to first token matters more than throughput. vLLM makes sense when you need to serve multiple concurrent requests, run a RAG application with many users, or get the most throughput from a fixed GPU budget. The two aren’t mutually exclusive — many teams prototype with Ollama and deploy with vLLM.

How does prefix caching work in vLLM and when does it actually help?

When enable_prefix_caching=True, vLLM computes a hash of each KV block’s token content. If a new request starts with a token sequence whose blocks hash-match previously computed blocks, those blocks are reused instead of recomputed. It helps most when requests share long identical prefixes — system prompts, static document context in RAG pipelines, or few-shot examples. It provides no benefit when every request has a unique prefix, and it can actively hurt if your prefix formatting is inconsistent, since small variations break the hash match.

What causes OOM errors even when gpu_memory_utilization seems conservative?

The most common cause is max_model_len set too high relative to your actual concurrency target. vLLM uses gpu_memory_utilization and max_model_len together to calculate how many blocks to pre-allocate at startup. If you set max_model_len=32768 but your actual sequences rarely exceed 4K tokens, you’re reserving blocks for headroom that never gets used — and the block pool ends up smaller than it could be. Set max_model_len to a realistic ceiling for your workload, not the model’s theoretical maximum.

---

Where to Go From Here

PagedAttention is what separates vLLM from naive inference wrappers — not just in benchmarks, but in the behavior you’ll see under real concurrent load. The block-table model, demand allocation, and copy-on-write for beam search are the mechanisms that make the throughput numbers real.

Start with the metrics endpoint. Watch gpu_cache_usage_perc under your actual load pattern before touching anything else. Then adjust one parameter at a time — max_num_seqs first, max_model_len second, block_size only if you have a specific reason. And if you’re running a RAG pipeline, turn on prefix caching. It costs nothing to enable and pays off immediately when your prompts are consistent.

That’s the whole playbook. The rest is just measurement.