Why buffering every log can make an incident worse
During an incident, every struggling service starts writing more error logs, just when you need them most. And that’s often the moment the logging platform starts rate-limiting what it accepts.
The natural instinct is to buffer everything until it recovers. Logs are evidence, so keeping all of them feels responsible.
But the buffer lives on the same machines that are already under pressure. As memory or disk fills up, the buffering process takes more CPU and application threads can end up waiting to write. The attempt to preserve the record starts making the incident worse.
So the real question is how to keep the evidence that matters without letting logging compete with the service it’s meant to explain. Let’s look at what that takes: priorities, bounded buffers, sampling and a plan for when the logging path itself is degraded.
Why log volume rises with failure
A central logging system is valuable because it makes distributed behaviour searchable. When an incident occurs, teams need to answer:
Retaining logs supports those questions. The catch is that log volume tends to rise with failure volume.
The event that creates the incident can also create the pressure that breaks the logging path. If the buffer has no bound, the logging strategy can consume the resources needed for recovery.
Not every log has the same value
Not every event needs the same delivery guarantee. A production system may emit:
Treating all of these as equally important creates an expensive and fragile pipeline. A better design classifies them:
Classifying events is not an excuse to lose important data. It is how the most important data survives when capacity is limited.
Buffers need an explicit failure policy
A buffer protects against short disruptions, but it also needs limits.
A bounded policy might look like:
The exact thresholds depend on the system; the important point is that “buffer everything forever” is not a design. It is an unbounded resource claim.
Logging must not block the primary workload
A service should not become unavailable because its logging destination is slow. For most request handling, logging should be asynchronous or non-blocking from the application’s perspective. A failed log delivery should not turn a successful user action into a failed one. There are exceptions; some security or financial workflows may require a durable audit record as part of the business operation. In those cases, the audit event needs a stronger transactional or outbox-style path, not merely an ordinary log statement. The distinction matters:
Calling both “logs” can hide the fact that they have different correctness requirements.
Repeated errors need aggregation, not amplification
An outage can produce thousands of near-identical stack traces. Writing every one of them in full may make the logging system slower and the incident harder to understand. A more useful approach is often:
This preserves the signal:
without requiring an operator to sift through thousands of identical messages. The goal is not to hide failures, but to keep failure evidence usable.
Structured logs make the pipeline recoverable
Free-form messages are readable but difficult to aggregate; a structured event can carry the fields needed for investigation:
{ "timestamp": "2026-10-05T18:00:00Z", "level": "error", "event": "subscription.webhook_processing_failed", "request_id": "req_123", "provider_event_id": "evt_123", "error_class": "database_timeout", "retry_attempt": 2}That makes it possible to ask useful questions:
It also reduces the temptation to log entire raw payloads.
Privacy is part of the logging design
Logs often become a place where sensitive information escapes ordinary data controls. Avoid recording:
A failure path deserves the same redaction discipline as a normal path. The worst time to discover that an exception logger captured raw request bodies is during an incident. Access to logs also needs governance; a powerful logging platform can become a broad internal data-access surface if anyone can search unredacted production records.
Final thoughts
Buffering every log feels safe because data loss is uncomfortable, but an unbounded logging buffer can convert a degraded observability system into degraded product availability. In practice that means classifying events, bounding buffers, protecting the primary workload, giving critical audit data a stronger path, aggregating repeated failures and making loss or delay visible.
Rather than promising never to lose a log, decide which evidence must survive, how to preserve it safely and how the product keeps running when the logging pipeline does not.