How a SaaS Team Reduced API Latency by 60% by Rethinking Their Node.js Backend Architecture

Eight months of steady growth, and then one enterprise customer onboarding nearly broke everything.

The team hadn’t changed the code. Traffic had just crossed a threshold where assumptions baked into the architecture — assumptions that looked fine at 500 req/min — started failing at 4,000. P95 latency went from 120ms to 800ms. Users noticed. Support tickets followed.

What they did next is worth studying. Three targeted changes to their Node.js backend — connection pooling, worker threads, and route restructuring — cut p95 latency by 60%, from 780ms to 195ms. No new infrastructure. No rewrite. This post traces each fix, the data that justified it, and the decision points that mattered most.

---

The Stack and Where It Started Breaking

The setup was a typical React and Node.js full stack: a Node.js 18 Express API serving the frontend, deployed on a single c5.2xlarge EC2 instance behind an Application Load Balancer. The team had followed sensible patterns. Nothing exotic.

Traffic was a mix of lightweight auth/session endpoints and heavier report-generation routes — CSV exports, PDF generation, data aggregations. For most of those eight months, this wasn’t a problem. The 15% month-over-month growth was gradual enough that no single week felt alarming.

But here’s what growth curves do: they hide compounding problems until they don’t.

Baseline metrics before any changes:

Metric	Value
p50 latency	95ms
p95 latency	780ms
p99 latency	1,400ms
CPU utilization (avg)	~40%
Event loop lag (peak)	300–500ms
DB connection errors	Frequent under load

CPU looked fine on average, which is why the team initially suspected the database. That instinct was partially right — just not for the reason they assumed.

What the Profiler Actually Showed

The diagnostic pass used three tools: clinic.js for event loop analysis, Node’s built-in --prof V8 profiler, and PostgreSQL connection pool logs. The combination told a two-part story.

First: the event loop was being blocked by synchronous CPU work during report generation. Those 300–500ms lag spikes weren’t I/O wait — they were the main thread doing real computation and starving everything else.

Second: the app was opening a new database connection for every request using new Client() directly. Under low concurrency, the TCP handshake cost is invisible. Under concurrent load, you hit PostgreSQL’s max_connections ceiling fast, and connections start queuing or dropping.

Two separate problems. Two separate fixes. Neither required touching the other.

---

Fix #1: Connection Pooling — 35% Latency Drop in a Single PR

This is the kind of fix that makes you feel slightly embarrassed in retrospect.

The pattern was everywhere in the codebase: route handlers importing Client from pg and calling new Client() at the top of each handler. Connection pooling is the practice of maintaining a set of pre-opened database connections that multiple requests share, rather than opening and closing a fresh TCP connection on every request. This codebase was doing none of that — every request opened, queried, and closed its own connection. Works fine locally. Collapses under real concurrency.

The fix was migrating to pg-pool with a centralized singleton:

// Before — scattered across route files
const { Client } = require('pg');
const client = new Client();
await client.connect();
const result = await client.query('SELECT ...', [params]);
await client.end();

// After — one pool for the entire process
const { Pool } = require('pg');
const pool = new Pool({
  max: 20,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 2000,
});
module.exports = pool;

One module. One pool. Every route handler imports it and calls pool.query(). Connections get reused. PostgreSQL’s connection ceiling stops being a concern.

They also added pool event listeners for connect, error, and remove — not for performance, but for observability. If the pool starts recycling connections abnormally, you want to know before users do.

Results after Fix #1:

• p95: 780ms → 510ms (35% improvement)
  
• DB connection errors: essentially zero
  
• Event loop lag: unchanged
  

That last point matters. CPU blocking was still there. The pool fix proved the remaining problem was compute, not I/O — which meant the next fix had to be different in kind, not just degree.

---

Fix #2: Worker Threads — Solving Node.js Backend Scalability at the Source

Here’s where most teams make the wrong call. They hear “event loop blocking” and immediately start drafting a migration plan — Go, Rust, microservices, whatever sounds architectural enough to justify the cost. This team didn’t. They profiled first.

The V8 profiler was unambiguous: the blocking was coming from three specific route handlers — CSV export, PDF rendering, and a monthly aggregation report. Everything else was fine. Rewriting the whole service to fix three routes would have been disproportionate.

Why worker_threads, Not child_process

Both let you run code off the main thread, but they’re not equivalent. Here’s how they compare:

	worker_threads	child_process
Shared memory	Yes (SharedArrayBuffer)	No
IPC overhead	Low	Higher
Process startup cost	Lightweight	Full Node.js process
Best for	CPU-bound computation	Isolated subprocess execution

worker_threads shares memory with the parent and has lower message-passing cost. For CPU-intensive work where you’re not spawning dozens of workers, it’s the better fit.

The team built a small WorkerPool class — not a library, just ~60 lines — that maintained a fixed pool of 4 threads (CPU count minus one), dispatched tasks via postMessage, and automatically replaced any worker that crashed.

// Usage from a route handler
const result = await workerPool.run('generate-csv', { reportId, filters });
res.setHeader('Content-Type', 'text/csv');
res.send(result);

The main thread queues the task and moves on. The worker handles the computation. Other requests don’t wait.

One thing the team explicitly did not do: they didn’t move I/O-bound routes to workers. Queuing a database query through a message channel adds overhead without benefit — the main thread is already good at I/O. Workers are for CPU. Profiling told them which was which.

Results after Fix #2:

• p95: 510ms → 310ms
  
• p99: 1,400ms → 480ms (the biggest single drop)
  
• Event loop lag: 300–500ms spikes → <20ms
  

The p99 improvement here was the most meaningful user-facing change. That tail was brutal for anyone hitting the service during a report generation cycle.

---

Fix #3: Route Restructuring — The 40ms Nobody Knew They Were Paying

Connection handling and compute were sorted. But the profiler still showed unnecessary overhead — overhead that had nothing to do with database connections or CPU work. The team found it by reading middleware timing logs.

Every request — including a simple health check or a lightweight session validation — was passing through 11 middleware layers. Body parsing, session deserialization, two logging middlewares, and an auth layer that hit the database on every single request, even for routes that didn’t need it.

For a route that should take 5ms, you were paying 40–60ms before any application logic ran.

The Restructuring Approach

The fix was splitting the Express app into two routers with different middleware stacks:

• publicRouter: health checks, static metadata, webhook receivers — minimal middleware, no auth overhead
  
• authRouter: everything that needs a verified session — full middleware stack, but with session lookups cached
  

The session cache used node-cache with a 60-second TTL. Session data doesn’t change mid-request, and a 60-second stale window is acceptable for nearly every SaaS use case. That single change eliminated the per-request database lookup for authenticated traffic.

Multipart parsing was moved to lazy initialization — only routes that actually receive file uploads mount that middleware. Everything else doesn’t pay for it.

The Decision Not to Rewrite

During this phase, someone on the team raised the question of migrating to Next.js API routes. The React vs Next.js for Enterprise Applications debate is real and worth having — Next.js colocation of frontend and API logic has genuine benefits for some teams. But the bottlenecks here were in specific route handlers, not in the choice of framework. Switching frameworks would have moved the same problems into different files.

The microservices question came up too. Same answer: the bottlenecks were localized. Microservices distribute problems; they don’t fix them.

Results after Fix #3:

• p50: 95ms → 38ms
  
• p95: 310ms → 195ms
  
• Middleware overhead on lightweight endpoints: 40–60ms → 6–8ms
  

---

Cumulative Results

Three changes. No infrastructure additions. No rewrite.

Metric	Before	After	Change
p50 latency	95ms	38ms	−60%
p95 latency	780ms	195ms	−75%
p99 latency	1,400ms	480ms	−66%
DB connection errors	Frequent	~0	Eliminated
Event loop lag	300–500ms	<20ms	−96%

The codebase is still a Node.js monolith. Still on one EC2 instance. The React.js frontend didn’t change — because it didn’t need to. All of this was server-side.

The sequence that guided the work matters as much as the fixes themselves:

1. Profile before touching anything
   
2. Fix I/O problems first — they’re the cheapest wins
   
3. Fix CPU blocking second — worker threads
   
4. Fix structural overhead last — middleware and routing
   
5. Consider a rewrite only if localized fixes don’t close the gap
   

---

Frequently Asked Questions

How do I know if my Node.js API has an event loop blocking problem?

The clearest signal is a widening gap between p50 and p99 latency. If your median response time looks fine but your tail latency is 10x worse, CPU blocking is a likely cause. Run clinic.js bubbleprof or use the --prof flag on a production-like load test to see where the main thread is spending time. A flamechart with wide synchronous blocks is the tell.

Is pg-pool the right solution for all Node.js database connection issues?

For PostgreSQL specifically, yes — pg-pool is the standard approach and it’s well-maintained. If you’re using other databases, most modern drivers have native pooling built in (Mongoose/MongoDB, Sequelize, Prisma). The principle is the same regardless of driver: one pool per process, sized to your database’s max_connections limit, reused across requests. Sizing matters — a pool of 100 on a database that allows 100 total connections leaves nothing for migrations or direct queries.

When should a JavaScript team actually consider breaking out of a Node.js monolith?

When profiling shows that the bottlenecks are distributed across the entire service rather than concentrated in specific routes. If fixing three route handlers gets you 60% improvement, you don’t need microservices. If every route is independently slow for different reasons and they’re competing for resources in ways that can’t be isolated, that’s when the calculus changes. The bar for splitting a monolith is higher than most teams set it.

Does using worker threads complicate deployment or debugging?

Somewhat. Worker thread errors don’t propagate automatically to the main thread — you have to wire that explicitly with worker.on('error', ...). And debugging a crash inside a worker requires extra tooling compared to single-threaded code. The operational overhead is real but manageable, and for any Node.js service with confirmed CPU-bound workloads, the tradeoff is worth it.

---

The Takeaway

Performance problems in Node.js are almost never mysterious — they’re diagnosable. The event loop model is genuinely good at what it was designed for. When it breaks down, it breaks down in predictable ways that profiling exposes.

If your team is watching p95 climb while p50 stays flat, start with clinic.js and your connection pool logs. Those two data sources will tell you whether you’re looking at an I/O problem or a compute problem, and that distinction determines everything that follows.

Fix the actual bottleneck. Profile again. Repeat until the numbers close.