Provider
BuildkiteIncident detail
Increased latency and error rates
Timeline window
to
Get alerted the next time Buildkite breaks
Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.
Timeline
Incident updates
Every update on the official status source, oldest to newest, exactly as it appeared there.
Investigating
We're observing increased latency and error rates due to an inability to scale up. We're currently investigating and will provide status updates as they become available.
Investigating
We're currently working on mitigations for scaling up, but at this stage service is degraded with increased latency across API, notifications, and builds starting.
Identified
We're continuing to see increased latency across much of our sub-systems due to an on going AWS outage. We are unable to launch new tasks in us-east-1 and are investigating potential mitigations to restore service.
Identified
We have implemented mitigations and see an improvement in latency for the Agent API. Latency and error rates continue to be elevated across Rest, GraphQL and Web service as well as notifications being delayed.
We are continuing to work through mitigations and will provide an update in 1 hour.
Identified
Our mitigations improved latency for the Agent API, although latency and error rates are still visible across other services. The us-east-1 issue is reporting some recovery and we are seeing further improvements in our services. We are actively monitoring the situation and implementing mitigations where possible.
Monitoring
We're seeing slow recovery of all our services. Latency and error rates are decreasing across the board. We are continuing to monitor the situation.
Monitoring
We're seeing signs of recovery across the board. Error rates have reduced to baseline levels. Latency is trending towards baseline.
We continue to actively monitor our services and the AWS reports on us-east-1 impact.
Monitoring
Latency and error rates have all returned to baseline levels. We have seen full recovery of our services.
We continue to actively monitor our services and the AWS reports on us-east-1 impact to ensure stability is maintained.
Resolved
Our services have been fully recovered for the last hour, so we are marking this as resolved.
Our engineers will continue to monitor AWS and will keep services scaled up to prevent impact from any additional failures.
Resolved
We have published a post-incident review here:
https://buildkite.com/resources/blog/post-incident-review-for-20th-october-2025/
Keep exploring
More from Buildkite
Neighboring incidents on Buildkite's timeline and the rest of their record on OutageDeck.