How to Build a Production-Ready AI Agent in Python Using LangChain

Most LangChain tutorials get you to “it works on my machine” — and stop there. That gap between a working prototype and something you’d trust in production is where teams quietly lose weeks of engineering time.

Building a production-ready agent isn’t about finding the right library. It’s about making deliberate choices around tool reliability, memory architecture, inference backends, and observability — then wiring them together in a way that holds up when real users hit it. This post gives you that blueprint: agent design, RAG-backed memory via LlamaIndex, local LLM development with Ollama, inference optimization with vLLM for production throughput, and deployment fundamentals.

---

Architecting the Agent: Tools, Reasoning, and Control Flow

The agent pattern you pick shapes everything downstream, so get this decision right early.

Which Agent Pattern Should You Use?

The three patterns you’ll encounter in LangChain are ReAct (Reasoning + Acting), OpenAI Functions-style structured calls, and fully custom loops. ReAct is the most general-purpose — the model reasons step-by-step before acting, which makes debugging easier. But it’s also the chattiest and least predictable under load.

OpenAI Functions (now “tool calling”) forces the model into a structured output format before invoking a tool. That constraint dramatically reduces the rate of malformed tool calls — which, if you’ve watched a naive ReAct agent hallucinate a function name that doesn’t exist, you’ll appreciate immediately.

I’d argue most production agents should default to tool calling. You give up some flexibility, but you gain consistency, and consistency is what lets you sleep at night.

Defining Tools the Right Way

Here’s a simple tool definition using LangChain and Pydantic:

from langchain.tools import tool
from pydantic import BaseModel, Field

class SearchInput(BaseModel):
    query: str = Field(description="The search query to look up")

@tool(args_schema=SearchInput)
def search_knowledge_base(query: str) -> str:
    """Search the internal knowledge base for relevant documents."""
    return retrieve_documents(query)

The args_schema is doing real work here. Without it, the LLM decides what arguments to pass — and at some point it will pass something your function can’t handle. The schema validates inputs before your code ever sees them.

Handling Failures Without Cascading Crashes

Production agents fail. The question is whether they fail gracefully or take down the whole request.

Set a max_iterations limit on your AgentExecutor — without it, a confused agent spins indefinitely. Add a fallback tool that returns a structured “couldn’t complete this” response rather than letting the agent hang. And wrap your tool implementations in explicit timeout logic; a 30-second hung tool call will quietly destroy your latency budget.

from langchain.agents import AgentExecutor

agent_executor = AgentExecutor(
    agent=agent,
    tools=tools,
    max_iterations=6,
    handle_parsing_errors=True,
    verbose=True,
)

---

RAG-Backed Memory Management for Production Agents

Short-term memory is easy. Long-term memory is where production agents get complicated.

Why Naive Memory Breaks Under Load

ConversationBufferMemory dumps the entire conversation history into every prompt. That works fine for a five-turn demo. It falls apart fast on long sessions — you’ll hit context limits, costs spike, and latency creeps up as the prompt grows. I’ve seen teams hit $0.30 per request on what should have been a cheap lookup, purely because they were dragging 80 turns of history into every call.

The fix isn’t to increase context limits. It’s to be selective about what memory actually needs to be there.

Building a RAG-Backed Memory Layer with LlamaIndex

RAG (Retrieval-Augmented Generation) is a memory architecture that fetches relevant documents from a vector store at query time, giving your agent access to external knowledge without bloating the context window on every call. Building this with LlamaIndex is one of the cleaner approaches I’ve come across — you index documents into a vector store, then retrieve only what’s relevant to the current query instead of stuffing everything into context.

Here’s how to wire a LlamaIndex retriever into LangChain:

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from langchain_community.retrievers import LlamaIndexRetriever

# Build the index
documents = SimpleDirectoryReader("./docs").load_data()
index = VectorStoreIndex.from_documents(documents)

# Wrap it for LangChain
retriever = LlamaIndexRetriever(
    index=index,
    query_kwargs={"similarity_top_k": 4}
)

The similarity_top_k parameter matters more than most people realize. Too low and you miss relevant context. Too high and you add noise that confuses the model. Four is a reasonable starting point; tune it against your actual queries.

Keeping Memory Clean Over Long Sessions

Two approaches worth knowing:

• Summarization buffers: LangChain’s ConversationSummaryBufferMemory keeps recent turns verbatim and compresses older turns into a summary — a good balance of recency and history.
  
• Session persistence: For multi-session agents, write memory to Redis (fast, low-durability) or SQLite (simple, durable). Redis is the better choice if you’re running multiple agent instances behind a load balancer.
  

---

Inference Backends: Ollama for Local Dev, vLLM for Production

Your inference backend choice affects cost, latency, and data privacy. Think through it before you’re in production — changing it later is painful.

Starting Locally with Ollama

Ollama is a local LLM runtime that serves open-source models through an OpenAI-compatible API, with no data leaving your hardware. Python local LLM development with Ollama has become genuinely pleasant for exactly that reason — you pull a model, start a server, and point LangChain at it with two lines of code:

from langchain_community.chat_models import ChatOllama

llm = ChatOllama(model="llama3.2", temperature=0)

Your agent code doesn’t change at all. This setup is exactly right for development and for workloads where data privacy matters — healthcare, legal, anything where you’d rather not send documents to a third-party API.

The tradeoff is hardware. An 8B parameter model in 4-bit quantization needs roughly 6GB of VRAM — a modern consumer GPU handles that fine.

Scaling with vLLM in Production

vLLM is an open-source inference engine that uses PagedAttention to manage GPU memory more efficiently than standard serving — treating the KV cache the way an OS treats virtual memory, which enables far more concurrent requests at the same hardware cost.

When you need throughput — multiple users, concurrent requests, low tail latency — Python LLM inference optimization with vLLM is the most practical option for self-hosted deployments. Start the server:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.2-8B-Instruct \
  --port 8000

Then point LangChain at it:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
    model="meta-llama/Llama-3.2-8B-Instruct",
)

Under 10 concurrent users, the difference between vLLM and a naive setup isn’t dramatic. At 50+ concurrent requests, vLLM’s throughput advantage typically runs 3–5x. At that scale, it’s not optional.

Backend	Best For	Hardware	Concurrency
Ollama	Local dev, privacy workloads	Consumer GPU	Low (1–5)
vLLM	Self-hosted production	Server GPU	High (50+)
Managed API	Low-ops production	None	Elastic

---

Deployment Fundamentals

Getting the agent to production means treating your inference backend and agent service as separate concerns.

Containerization That Actually Works

Run your agent and your inference backend in separate containers. Your agent container is stateless and scales horizontally; your vLLM container is GPU-bound and scales vertically. Mixing them into a single image makes both harder to operate.

Keep model endpoints and API keys in environment variables, not code. That’s basic — but I still see it hardcoded in repos that should know better.

Observability Is Not Optional

You can’t debug a blind agent. Wire up LangSmith (LangChain’s native tracing) or push spans to OpenTelemetry. At minimum, you want visibility into which tools were called, in what order, how long each took, and how many tokens each step consumed.

The signal that matters most in practice is tool call failure rate. When it spikes, you’ve either got a prompt regression or a tool reliability issue. Without tracing, you can’t tell which.

Rate Limiting and Output Validation

Set token budgets per request and per user. An unbounded agent can easily spend 10x your expected token count on a single bad query. A simple post-processing validation step — or a library like Guardrails AI — catches malformed outputs before they reach your users.

---

Frequently Asked Questions

Is LangChain still the right choice for production agents?

LangChain has matured considerably — LangGraph in particular gives you explicit state machines for agent control flow, which is far better for production than the older AgentExecutor pattern. It’s a reasonable foundation, but expect to write custom logic around its edges rather than expecting the library to solve every problem out of the box.

How do I choose between Ollama and vLLM?

Use Ollama for development, prototyping, and privacy-sensitive workloads where data can’t leave your infrastructure. Switch to vLLM when you need to serve more than a handful of concurrent users or when consistent latency matters. They both expose OpenAI-compatible APIs, so switching between them is a one-line config change in your LangChain setup.

What’s the biggest mistake teams make with their first production agent?

Skipping observability. Teams build an agent that works great in testing, deploy it, and then have no idea why it behaves differently in production. Trace every agent run from day one — it’s far cheaper to add tracing before launch than to debug blind after something goes wrong.

Can I use LlamaIndex and LangChain together?

Yes, and many production systems do exactly this. LlamaIndex is particularly strong for document ingestion, indexing, and retrieval. LangChain handles agent orchestration, tool management, and the execution loop. Think of LlamaIndex as the retrieval layer and LangChain as the agent layer sitting on top of it.

---

The Architecture Holds — Your Domain Logic Is What Changes

The teams that ship reliable AI agents aren’t using magic libraries. They’re making deliberate decisions at each layer and handling failures explicitly.

Start with tool schemas and iteration guards. Add LlamaIndex-backed retrieval when naive memory breaks. Develop locally with Ollama, then switch to vLLM when you need scale. And trace everything from day one.

Clone a working reference implementation, swap in your own tools, and build up from there. What you’ll iterate on is the tooling and prompt logic specific to your problem — not the underlying architecture.