When CPU looks fine but requests still stall
Some systems don’t crash under load. They stall. Pages that were fast all morning start hanging once lots of people sign in at the same time. Some requests eventually load, others time out, and restarting a server only helps for a while.
But CPU isn’t maxed out and the database isn’t pinned at 100%, so the natural first step is to profile the code and look for the slow line.
Profiling helps when a function is doing expensive work. When the slowdown only appears as requests overlap, though, the time is usually going somewhere a CPU profile can’t see: requests waiting for a connection, a lock, a queue or another service.
So instead of asking which line of code is slow, it’s more useful to ask where the work is waiting. Let’s follow that waiting time through a request and see what it tells you.
Where the time actually goes
A profile will reveal expensive JSON processing, a huge response, an unindexed query or repeated external calls. A peak-only slowdown is more deceptive, because a request can take over nine seconds while doing very little work. For example:
The endpoint is slow, but the business logic is not necessarily the cause. A CPU profiler samples only active work, so it shows almost none of those nine seconds; a wall-clock or off-CPU view, or a trace with timings around each wait, is what shows where the time went.
Waiting accumulates before failure appears
Every system has finite resources:
When demand approaches a limit, work begins to wait. At first, the waiting may be small enough that nobody notices. As utilisation increases, the queue can grow quickly; a small increase in traffic can create a large increase in tail latency because more work is competing for the same resource. That is why a typical request can look fine while real users are struggling. For example, the median can stay healthy while the tail does not:
The median, like an average, hides the fact that a meaningful part of the user population is experiencing a broken workflow.
Start with the request timeline
When a system stalls, I want to split total request time into components:
Then I want to compare the incident window with normal traffic. A useful trace might show:
That points toward a connection or concurrency issue, not a suddenly slow query.
Another trace may show connection wait and query time much as they are under normal traffic, while the call to the external provider accounts for most of the request. Here the database is barely involved. The provider may be slowing under its own load or throttling the concurrent calls the service now sends it. The same user-visible symptom leads to a different investigation, and nothing on the application's own CPU graph would have pointed there.
Traces have limits of their own. They only show the waits someone instrumented, so time spent queuing for a pool or a lock without its own span appears as an unexplained gap, or not at all. Sampling can also drop the slow requests being investigated, particularly when the decision to keep a trace is made before anyone knows the request will be slow. No slow span does not mean nothing was slow.
Restarting the service
Restarting can reduce pressure temporarily, and it may clear local queues, release stuck resources, drop in-flight work, reset a connection pool, or remove a bad instance. During an incident, that can be a legitimate mitigation. But it is not an explanation. If the pressure is created by demand exceeding a shared downstream resource, the system will fill up again after the restart. A restart can also destroy useful evidence:
When possible, capture the state first:
- Which routes are slow?
- Which resource has increasing wait time?
- Are retries rising?
- Which dependency changed first?
- Is queue depth growing?
- Are lock waits present?
- Which deployment or configuration changed recently?
The goal is to restore service without losing what the incident could have taught you.
Retries can turn a stall into a feedback loop
Stalled systems often receive more traffic because users and clients react to uncertainty. A person refreshes, a browser retries, a mobile client reconnects, a job worker retries and a load balancer sends another attempt.
The system ends up processing the original work plus the retries, which is why retries need:
A retry is not inherently a reliability feature; under the wrong conditions, it is an amplifier.
What is full?
A productive incident investigation asks:
The answer may be:
Each problem has a different response, and a longer timeout fixes none of them. The fix has to change the constrained part of the system.
While that fix is found, pausing non-critical work such as email sending, analytics updates or report generation can keep sign-in and core reads and writes moving. Which work counts as critical is a product decision, and it is easier to make before an incident than during one.
Start where time accumulates
A peak-traffic stall is rarely one slow function. More often it is the interaction of finite capacity, concurrency, queues, retries, dependencies, and work that stayed synchronous longer than it needed to. When it happens, do not start by guessing which limit to increase. Start by finding where time accumulates. Then ask: What is waiting, what is it waiting for, and what can we change so the important work continues to move?