Buildkite service disruption

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

We are continuing to investigate this issue. We are seeing impact on the Agent API which will affect job scheduling, artifact uploads, and an increase in 5xx responses from the Agent API endpoints.

We are continuing to investigate elevated error rates across multiple services, and are working to determine the cause.

We are seeing improvements across the affected services, and are seeing services return to normal functionality. We are continuing to monitor and are determining the root cause.

We have seen full recovery for customers since 20:28 UTC.

We experienced an autoscaling feedback loop which increased the number of connections to our redis cluster above its ability to respond. This had widespread impact for all of our customers with Web UI, Agent API, REST API and job queue impact between 18:43-19:21 UTC, and again between 20:02-20:28 UTC. A full post incident review will be available later this week.

## Service Impact

_All times UTC unless stated otherwise._

Customers experienced elevated errors and latency across the Buildkite web interface, REST API, Agent API, and job queue during two periods: 18:43:00–19:21:00 and 20:02:00–20:28:00 on September 9, 2026.

During these periods, customers encountered failed API requests, delayed job dispatch, and errors in agent operations including authentication, job acceptance, and artifact uploads. Some jobs impacted by the incident did not recover automatically. These jobs had to be manually cancelled and retried to complete successfully.

Between 18:43:00–19:21:00: Agent API had an error rate up to 11.2% with no latency impact. REST API had a 9.7% error rate with no latency impact. Web request had a 2.6% error rate with no latency impact. Job queues were delayed by up to 3 minutes.

Between 20:02:00–20:28:00: Agent API had an error rate up to 24.7% with no latency impact. REST API had a 4.9% error rate with no latency impact. Web requests had a 0.8% error rate with no latency impact. Job queues were delayed by up to 10 minutes. During this window, notifications were also delayed up to 1 minute.

Core services fully recovered by 20:28:00.

## Incident Summary

### Background

Several Buildkite services use a shared Redis cluster for coordination, caching, and agent-facing operations. This is one of the last pieces of non-shard aligned shared infrastructure, and is on our roadmap to address. AWS enforces a connection limit of 65,000 connections per node on this Redis cluster.

As part of our ongoing migration from Amazon ECS to Amazon EKS, we reduced the number of Ruby threads in each Puma and Sidekiq process to their framework defaults. In ECS, we historically configured high thread counts, which caused resource contention and increased latency without a corresponding increase in throughput.

We’d intentionally timed this work as part of the EKS migration because the migration already required us to redesign how these services are sized and scaled: EKS can respond directly to request queues and worker utilization, allowing us to run more, smaller application containers while preserving total capacity. This gives customers faster, more predictable API responses and background-job processing, particularly during periods of high load.

### What happened

We began testing Puma Agent services on EKS in July and migrated traffic incrementally from late July, completing the migration earlier this week. Running more, smaller processes increased total Redis connection demand—a trade-off we had modeled, but not adequately accounted for before completing the migration.

Before the incident, our shared services were already using much of the available connection capacity on the impacted Redis cluster, around 60-80% throughout a 24 hour window.

At 18:43:10 we started seeing elevated HTTP request errors. At this time, our application reported a burst of Redis connection timeouts, all targeting a single replica in the cluster. Clients with active Redis connections then attempted to reconnect. Failed connections were retried, causing a surge in connection attempts that exhausted that same Redis host’s connection-tracking allowance. This caused the host to reject new connections from the Agent API’s application pods, which led to increased request latency for our Puma Agent service.

At 18:43:30, the increased request latency from this network failure caused our application platform to rapidly start more Puma Agent containers to respond to the increase in request latency. Within a minute, we’d doubled our running pod count. Each additional container opened its own pool of Redis connections.

At 18:44:00, several Redis nodes hit their limit of 65,000 connections, and latency of requests slowed further triggering more Puma Agent autoscaling.

Once the Redis connection limits were reached, applications could no longer reliably connect to Redis. This caused failures across agent authentication, job assignment, artifact uploads, and other API operations.

The failures then created a feedback loop: slower requests caused our shard aligned Puma Agent services to autoscale, and those additional containers opened more Redis connections, placing further pressure on Redis.

By 18:50:00, our system stabilized at max, running about five times the number of pods we’d started with. This stable state allowed the autoscaling to start bringing pod counts back down.

By 19:10:00 the error rate had returned to 0 and and services were no longer degraded.

By 19:20:00 our pod counts had autoscaled down to more reasonable but still high numbers, and the Redis cluster was functioning again. At this point customers saw recovery.

At 20:02:00, a routine application deployment temporarily introduced additional containers. Since the Redis cluster was already under strain, this quickly increased request latency which added autoscaling on top of the regular deployment surge numbers. This amplified the connection growth again and caused the second period of degraded service, lasting until 20:28:00.

The influx of new pods resulted in a very large number of Envoy Gateway configuration updates, which in turn caused some network throttling of the underlying node. This caused packet drops and delays updating our Cilium Operator, which controls configuration for the cluster’s overlay network. This caused resulted some Buildkite Jobs to remain in a stuck state for some customers, requiring manual retries.

### How we responded

When we saw that the Redis cluster had reached its capacity to handle incoming connections, we immediately began adding more nodes to the cluster, increasing incoming connection capacity by 33%.

When we saw the impact of deployments, we immediately paused further application deployments while we investigated the connection growth.

We increased the Puma Agent utilization autoscaling threshold, reducing the rate at which workloads scale in response to brief load increases. We also increased our scale up stabilization window to prevent future runaway autoscaling of our workloads.

We reduced the configured maximum surge of our workloads during application deployments, to reduce pod churn impact on connections.

Since Redis connections and latency remained stable, we then resumed deployments while continuing to monitor the cluster.

## Changes we're making

Reduce autoscaling sensitivity.

• We increased the utilization threshold that triggers the Puma Agent autoscaling, so short-lived latency increases do not cause rapid increases in application and Redis connection demand.
• We increased our scale up stabilization window so that we only add more pods in response to scaling triggers in incremental bursts. This will throttle scaling up based on downstream latency and runaway scaling.

Reduce deployment-related load.

• We reduced the maximum deployment surge for Puma and Sidekiq workloads, limiting the additional capacity and downstream connections introduced during deployments.

Increase immediate Redis capacity by 33%.

• We added more nodes to the Redis cluster to handle more connections, given the shape of our connection load has shifted with our different thread counts in EKS

Reduce connection pressure from high-traffic workloads.

• We moved the two largest Puma Agent sharded deployments back to ECS while we evaluate how the threading changes impact Redis connections and our autoscaling sensitivity.

Bound aggregate Redis connection demand.

• We are reviewing Redis connection-pool sizing, application thread counts, and workload scaling limits so that application growth cannot exceed the available Redis connection capacity.