Checkout p99 after a harmless-looking deploy

The alert named checkout. Checkout was healthy.

At 03:17 UTC, the synthetic checkout service crossed its two-second p99 alert threshold. Error rate followed ninety seconds later. A payment-service release had completed twenty-four seconds before the first latency change, but its canary checks were green and its application code contained no query changes.

The tempting conclusion was “bad deploy.” That was directionally useful and operationally incomplete. A deploy is an event, not a root cause. The investigation still had to identify the changed mechanism, its downstream effect, and the evidence connecting it to the customer-facing symptom.

The service named in the alert was two dependencies away from the change that caused it.

Signal packet

All values below are authored for this simulation. They are deliberately plausible, not measured Tiravan or customer results.

Follow latency, not ownership.

1. Establish the symptom boundary

Checkout CPU, memory, and request concurrency remain inside their previous-hour bands. That does not clear checkout, but it removes the simplest resource-exhaustion explanation. The p50 remains stable while p99 climbs sharply, which points toward queuing on a subset of requests rather than uniformly slower application work.

2. Reject the obvious database story

Database CPU is 41%, query latency is flat, and no lock-wait increase appears. The phrase “database latency” would be technically defensible because requests are waiting on the database path. It would also misidentify the mechanism: they are waiting for a client connection before a query begins.

3. Walk the slowest trace leaf

The trace tree moves responsibility from checkout-service to payment-service, then from payment business logic to its connection-acquisition span. Loki confirms the same boundary independently: active connections remain pinned at twelve while waiters rise from zero to eighty-seven.

4. Connect the mechanism to the change

The deploy contains no slow query and no new checkout code. It does contain one configuration change that exactly explains the observed ceiling of twelve active connections. The canary did not catch it because its traffic never required more than eight concurrent database connections.

Causal Chain RCA

1. The release reduces the payment-service connection pool from 40 to 12.
2. Production concurrency exceeds the new cap; acquisition waiters accumulate.
3. Payment spans spend up to two seconds waiting before their queries begin.
4. Checkout inherits the downstream delay and crosses its p99 alert threshold.
5. Acquisition timeouts turn the latency incident into customer-visible failures.

{
  "summary": "checkout p99 rose after payment connections queued",
  "trigger": "payment-service DB_POOL_MAX changed from 40 to 12",
  "effect": "connection acquisition wait exceeded 2 seconds",
  "root_cause": "production concurrency exceeded the reduced pool cap",
  "evidence_cell_ids": [
    "E-0142",
    "E-0147",
    "E-0151",
    "E-0158"
  ]
}

Rolling the configuration back to forty drains the queue and restores p99. That is useful confirming evidence, but it should not replace the preceding chain: “rollback fixed it” proves correlation with the release, not which part of the release mattered.

What an on-call engineer should carry forward

• Separate connection acquisition time from query execution time in both traces and metrics.
• Load-test configuration defaults, not only application code.
• Treat the alerting service as the symptom boundary, not the presumed owner.
• Use rollback as confirmation after identifying the mechanism—not as the entire RCA.