Benchmarking Python LLM Inference in 2025: What the Numbers Actually Say About vLLM

A team I consulted with last year spent six weeks tuning their retrieval pipeline, convinced the vector search was killing their RAG application’s response times. When we finally profiled end-to-end, the retriever was taking 40ms. The inference layer was taking 4,200ms. They’d been optimizing the wrong thing entirely.

This post focuses on Python LLM inference optimization — with real benchmark data, not marketing claims — comparing vLLM, Ollama, and llama.cpp across throughput, latency, and GPU memory efficiency on 7B, 13B, and 70B models. By the end, you’ll know exactly which backend fits your workload and why.

---

What Python LLM Inference Benchmarks Actually Measure (And Why Most Lie)

The Three Numbers That Matter in Production

Not all latency is equal, and throughput means nothing without context. The metrics that actually determine whether your inference layer survives real traffic are:

• Throughput (tokens/sec under concurrent load) — not peak single-request throughput, which is mostly useless at scale
  
• Time-to-first-token (TTFT) is the elapsed time between submitting a request and receiving the first output token — the latency your users actually feel; in streaming responses, a 500ms TTFT feels broken
  
• GPU memory efficiency — directly controls how many models you can serve per GPU, and therefore your infrastructure cost per query
  

Why Comparison Posts Usually Get This Wrong

Here’s the problem with most published LLM inference benchmarks: they measure single-request throughput. One request, one response, best case. That’s not a server — that’s a desktop app.

At 1 concurrent request, llama.cpp often beats vLLM on TTFT. At 16 concurrent requests, vLLM is outputting 3–4x as many tokens per second. These two facts can coexist, and ignoring the concurrency dimension makes benchmarks actively misleading.

For this analysis, requests were sent at controlled concurrency levels (1, 4, 16, 32) with a fixed 512-token output length, temperature 0.8, and a 10-request warmup before measurement. Hardware: A100 80GB for the main results, with spot checks on RTX 4090 for the 7B models.

---

vLLM: What PagedAttention Actually Does to Your Numbers

The Architecture Change That Changes the Results

PagedAttention manages the KV cache the way an OS manages virtual memory — allocating GPU memory in fixed, non-contiguous pages on demand. That eliminates the fragmentation that plagues static KV-cache allocation, which means the engine can schedule far more requests into the same VRAM budget.

Pair that with continuous batching — where new requests enter an active batch as soon as a slot frees, rather than waiting for the whole batch to finish — and you get a system built specifically for concurrent request handling. Static batching holds everything until the longest sequence completes. That’s a real problem when output lengths vary, which they always do in production.

The AsyncLLMEngine puts all of this within reach from Python:

from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams

engine = AsyncLLMEngine.from_engine_args(
    AsyncEngineArgs(model="meta-llama/Llama-3-8B-Instruct", max_model_len=4096)
)

async def generate(prompt: str) -> str:
    params = SamplingParams(temperature=0.8, max_tokens=512)
    async for output in engine.generate(prompt, params, request_id="req-1"):
        result = output
    return result.outputs[0].text

The Throughput Numbers on Llama-3-8B (A100 80GB)

Concurrency	vLLM (tok/s)	Ollama (tok/s)	llama.cpp (tok/s)
1	87	79	94
4	312	198	201
16	891	341	298
32	1,240	389	304

At low concurrency, the differences are marginal. By 32 concurrent requests, vLLM is outputting more than three times the tokens per second of either alternative.

That’s not a rounding error. That’s a fundamentally different system under load.

VRAM Efficiency at 70B Scale

At Llama-3-70B on an A100 80GB, vLLM loads and serves with approximately 68GB peak VRAM utilization under moderate load. llama.cpp with Q4_K_M quantization fits the same model into roughly 42GB — viable on smaller GPUs where FP16 won’t load at all. You get lower memory pressure with llama.cpp, but you trade it for concurrency headroom. Whether that’s the right call depends entirely on your hardware constraints.

---

Ollama and llama.cpp: An Honest Look

Ollama Is the Right Tool for Local Development (And That’s Fine)

Python local LLM development with Ollama is genuinely pleasant. One binary, an OpenAI-compatible REST API out of the box, and a model library that covers 90% of popular models. The Python client is three lines:

import ollama

response = ollama.chat(
    model="llama3.2",
    messages=[{"role": "user", "content": "Explain KV cache in 50 words"}]
)
print(response["message"]["content"])

The problem shows up when you move from development to any kind of production simulation. Ollama’s concurrency handling plateaus early — throughput at 16 concurrent requests was 38% of vLLM in testing on the same hardware. The culprit is the absence of continuous batching. Requests effectively queue rather than interleave.

For a single developer iterating on prompts and system messages, this doesn’t matter. For anything serving multiple users simultaneously, it does.

llama.cpp: The Quantization Argument Is Real

Q4_K_M quantization is legitimate. Running Llama-3-70B in 4-bit on hardware that can’t hold FP16 weights isn’t a compromise — it’s the only option, and llama.cpp handles it well. Single-request TTFT is often 10–20ms better than vLLM at small model sizes, because vLLM carries real per-request scheduling overhead. At concurrency = 1, you might genuinely prefer it.

But the throughput ceiling is hard. At 32 concurrent requests, llama.cpp saturated around 304 tok/s and stopped scaling. vLLM at the same load was still climbing.

Side-by-Side: Pick Your Column

Metric	vLLM	Ollama	llama.cpp
Setup complexity	High	Low	Medium
Single-request TTFT (7B)	~110ms	~120ms	~95ms
Throughput @ 32 concurrent	✓✓✓	✓	✓
Quantized 70B (48GB GPU)	Needs AWQ/GPTQ	Partial	✓
Python integration	AsyncLLMEngine / OpenAI API	OpenAI API	llama-cpp-python
Production-ready	Yes	With caveats	With caveats

---

Where This Plays Out: vLLM in a Python RAG Pipeline

Wiring vLLM Into LlamaIndex

Building a Python RAG application using LlamaIndex and wanting vLLM’s throughput behind it? The integration is straightforward with the Vllm class from llama-index-llms-vllm:

from llama_index.llms.vllm import Vllm
from llama_index.core import VectorStoreIndex, Settings

Settings.llm = Vllm(
    model="meta-llama/Llama-3-8B-Instruct",
    vllm_kwargs={"max_model_len": 4096, "gpu_memory_utilization": 0.90},
)

index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What does the document say about X?")

In a LlamaIndex query pipeline with 10 concurrent users, switching from Ollama to vLLM reduced median end-to-end response time from ~5.8s to ~2.1s in one production migration. Retrieval from a Pinecone index held steady at ~60ms either way. All the latency was sitting in generation.

Here’s the thing: most teams profile their retriever first because it’s the piece they built. The inference layer arrives as someone else’s library and gets treated as a constant. It’s not.

The Concurrency Crossover Point

Below 8 concurrent users, the backend choice rarely matters enough to justify vLLM’s operational overhead. Above 16, the gap widens fast enough that sticking with Ollama or llama.cpp starts showing up in user-facing latency.

That crossover is hardware-dependent. But on A100-class hardware, it falls consistently around 8–12 concurrent requests in these benchmarks. If your traffic never exceeds that ceiling, optimize elsewhere — inference backend selection shouldn’t be in your top five priorities.

---

Frequently Asked Questions

Is vLLM faster than Ollama for all use cases?

No — and this is where many comparisons mislead. At low concurrency (1–4 simultaneous requests), the performance difference is small, often under 15%. vLLM’s advantages compound at higher concurrency through continuous batching and PagedAttention. If you’re building a single-user tool or doing development work, Ollama’s setup simplicity outweighs any raw throughput edge.

Can I run a 70B model with vLLM on a single GPU?

Only with quantization or a large card. Llama-3-70B in FP16 requires approximately 140GB of VRAM — two A100 80GB GPUs at minimum with tensor parallelism enabled. vLLM supports AWQ and GPTQ quantization, which can bring that under 80GB on a single card. llama.cpp with Q4_K_M fits 70B into roughly 42GB, making it the practical choice on a single 48GB GPU if you can accept the concurrency ceiling.

Does switching inference backends break my existing Python application?

Generally not. Both vLLM and Ollama expose OpenAI-compatible APIs, so if your application already uses the openai Python client, pointing base_url at either backend requires no other code changes. The differences surface in advanced features — streaming behavior, logprobs, token limits — where the APIs diverge slightly. For LlamaIndex specifically, swapping the LLM provider is a one-line change in Settings.llm.

How quickly do these benchmark numbers go stale?

Faster than most people expect. Ollama added significant concurrency improvements in the 0.3.x series, and vLLM’s AWQ quantization support has reduced its memory footprint substantially since mid-2024. The relative ranking at high concurrency has held steady, but absolute numbers shift with every major release. Run your own benchmarks before making infrastructure decisions — the methodology here is reproducible, and your numbers will differ from these.

---

The Actual Recommendation

Pick your backend based on your concurrency profile, not on which benchmark post you read last.

Ollama for local development: yes, unambiguously. vLLM for production serving above ~10 concurrent users: yes, the data supports it clearly. llama.cpp when your GPU can’t hold the full-precision weights: the GGUF format is your only real option at that scale, and it works.

The team from the opening story ended up running vLLM behind a FastAPI server for production and Ollama locally for development — same model, OpenAI-compatible API, different backends for different load profiles. That split was the right call, and getting Python LLM inference optimization right with vLLM ultimately came down to one thing: they measured their concurrency profile before they chose a backend.

Profile before you optimize. The inference layer is often the bottleneck — but measure it first.