Deployment Degradation – Failures in Prod1,2,3

Deployment Degradation – Failures in Prod1,2,3

We are continuing to investigate this issue.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

### Incident Summary

On May 6 at 11:50 PM PST, we deployed a configuration change to one of our core pipeline

services. This change introduced an unintended interaction with our database layer, causing a

significant increase in write load. The resulting pressure degraded query and command

throughput across the platform

### Root Cause

The configuration change introduced a blocking condition on expression evaluation in the

pipeline service. When executions encountered blocked expressions, they failed and retried

repeatedly, generating a write storm against the database and the throughput went 4x

‌

### Remediation

● Rolled back the configuration change.

● Applied database-level tuning to reduce write pressure and accelerate backlog drainage

● Performed a controlled failover to a healthy database node to restore throughput

● Scaled up database nodes to provide sufficient capacity for full recovery

‌

### Preventive Actions

To prevent from such Issues happening again, we are focussing on:

1. Increased database resilience: We are implementing automated load-shedding thresholds

that trigger on leading indicators (replication lag, session depth, op latency) before the database

reaches saturation, preventing retry storms from compounding into full degradation events.

2. We are optimizing databases so that we can increase write throughput by an order of magnitude and enable independent scaling of customer data workloads. This would have allowed us to drain the message backlog nearly instantaneously during this incident.