2026-July-16 Resolved Service Incident - Error Reporting (Backtrace)

Between July 16th 13:03 UTC and July 16th 17:33 UTC, we experienced a service disruption where symbol archives utilizing multi-part upload protocols failed to process. Incident has been resolved and all services are operational.

### Dates:

Thursday July 16th 2026, 13:03 – 17:33 UTC

### What happened:

Symbol archives uploaded in multiple parts failed to process. All regions were affected.

### Why it happened:

Our symbol processing service sent an upload-verification field that our cloud storage provider's API does not accept for multi-part uploads, so those uploads were rejected. A fix for this had already been developed, but it had not yet been included in a released build and the service was not configured to use it. As in the first incident, rejected uploads were retried and accumulated on local disk.

### How we fixed it:

We deployed a build containing the fix and corrected the service configuration on the affected workers. A large backlog of uploads then processed, which briefly re-filled the disks before draining completely.

### What we are doing to prevent it from happening again:

We are releasing the fix formally through our build pipeline and persisting the corrected configuration in our configuration management, so it can't be lost. The disk-utilization alerting added after the first incident also covers the disk-exhaustion pattern common to both.