Latency on Pipelines

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

We're experiencing latency with processing job dispatch, pipeline uploads and incoming webhooks. We have identified the cause and are rolling forward with a fix now.

We've identified that the impact is more widespread that we initially understood - scheduled builds as well as job dispatch. The fix that is rolling out now is expected to address all impact.

The fix has been fully deployed and we are seeing signs of recovery. All events impacted by this issue will be automatically retried. We are continuing to monitor the fix for stability.

We've confirmed all impact from this incident has been resolved.

## Service Impact

Between 04:40 and 05:34 UTC on 3 August 2026, Buildkite stopped running recurring background work. Every background job process across the product was affected. The customer-visible impact was concentrated in Pipelines.

During this window, scheduled builds did not start at their scheduled time, jobs waited longer than usual before being dispatched to an agent, in some cases by more than ten minutes, and pipeline uploads and inbound webhook processing were delayed. Job timeouts were not enforced, so a job that should have timed out kept running, and agents that had finished or gone away were not cleaned up promptly.

Scheduled builds were not skipped. Every schedule that came due during the window ran once background scheduling recovered, up to around forty minutes late. A schedule that fires more often than that ran once on recovery rather than once for each interval it missed.

No builds, jobs or queued work were lost. Work that was delayed was processed once the incident was resolved, and jobs that errored were retried automatically. The dashboard, the REST and GraphQL APIs, and agents already running jobs were not affected.

## Incident Summary

At 04:33 UTC we merged a routine update to Rails, the web framework Buildkite runs on, taken to stay current with a security patch. Alongside Rails, that update pulled `connection_pool` from 2.5.5 to 3.0.2. The 3.0 release made that library's API keyword-only, which changes how every caller has to invoke it. Our review of the change covered our own code's use of `connection_pool` and updated it accordingly. It did not cover the other gems in our bundle that call the same library.

One of those is Sidekiq, which runs our background jobs. From 04:40 UTC, as the deploy rolled out, three internal Sidekiq threads inside every worker process failed on startup and stopped. The processes themselves stayed healthy and kept working jobs that were already queued, so nothing crashed and no deploy failed. What stopped was the scheduling of recurring work, which is what triggers scheduled builds, dispatches queued jobs to agents, enforces job timeouts and cleans up finished agents. That scheduling currently runs through a single elected process for the whole background job fleet, so it stopped for every product and every customer at the same moment.

Our monitoring alerted us at 04:48 UTC and we identified the change responsible at 04:59. Our first assessment was that there was no customer impact, because everything we could see was still moving. It took another ten minutes to recognise that recurring work had stopped across the whole product, at which point we published a status page and raised the severity. A fix pinning `connection_pool` back to 2.5.5 was merged at 05:10 and deployed ahead of the normal merge queue. Scheduling resumed at around 05:19 and the incident was resolved at 05:34.

## What contributed to this happening

`connection_pool` is called both by our own code and by gems we depend on. Our review, and the automated review tooling that flagged the change on the pull request, both looked at how we call it and stopped there.

Nothing in the dependency graph prevented the combination. Sidekiq 7.3.9, the version we run, permits any version of `connection_pool` from 2.3.0 upwards. Sidekiq 7.3.10 added an upper bound that would have refused 3.0, but we had not adopted it.

The failure only appears when a real Sidekiq process starts up. Our test suite does not exercise that path, and our pre-production check verifies that the application boots rather than that background scheduling is still running.

Because recurring work is scheduled by a single elected process for the whole fleet, one failure removed it for every customer at once rather than for a single group of customers.

## Changes we're making

We have pinned `connection_pool` to a version compatible with the Sidekiq release we run. This shipped as part of resolving the incident.

We are upgrading Sidekiq to 7.3.10, which declares an upper bound on `connection_pool` and so makes this combination impossible to resolve rather than something we have to remember to avoid.

We are changing how recurring work is scheduled so that it is no longer driven by a single process for the whole fleet. Each group of shards will schedule its own recurring work independently. This would not by itself have prevented this incident, because the incompatible library was loaded by every process, but it removes a shared point of failure that can currently stop recurring work for every customer at once. This work is already in progress.

We are updating our internal engineering guidance so that an upgrade moving a shared library to a new major version is reviewed against every consumer of that library, including the other libraries that depend on it, rather than only our own code.