{"meta":{"version":"v1","pricing":{"public":{"label":"Public","description":"Read-only API access for lightweight status checks and public integrations."},"premium":{"label":"Premium (early access)","description":"API keys with higher hourly quotas are available now; webhooks, alerting, and exports are on the roadmap."}},"generatedAt":"2026-07-07T22:57:10.600Z"},"data":{"id":"incident_statuspage_circleci_rp9y8kszn25j","slug":"circleci-delays-starting-docker-jobs-2026-06-23","title":"Delays starting Docker jobs","summary":"Delays starting Docker jobs","status":"resolved","severity":"major","startedAt":"2026-06-23T18:34:35.087+00:00","updatedAt":"2026-06-27T00:16:07.95+00:00","resolvedAt":"2026-06-23T19:58:24.368+00:00","provider":{"slug":"circleci","name":"CircleCI"},"affectedServices":[{"slug":"circleci-jobs","name":"Job Execution"}],"links":{"html":"/incidents/circleci-delays-starting-docker-jobs-2026-06-23","api":"/api/v1/incidents/circleci-delays-starting-docker-jobs-2026-06-23","providerHtml":"/providers/circleci"},"impactSummary":"CircleCI reported a major event for the affected tracked services.","source":{"id":"source_circleci_status","kind":"official_api","name":"CircleCI Status","checkedAt":"2026-07-07T22:55:01.749+00:00","officialUrl":"https://status.circleci.com","statusPageUrl":"https://status.circleci.com"},"updates":[{"id":"update_statuspage_circleci_rp9y8kszn25j_tgq5vrgwhfyr","status":"investigating","body":"We are investigating reports of delays starting Docker jobs.","createdAt":"2026-06-23T18:34:35.183+00:00"},{"id":"update_statuspage_circleci_rp9y8kszn25j_0z2ckx7q3q0j","status":"identified","body":"We are deploying a change to mitigate this issue.","createdAt":"2026-06-23T18:42:28.562+00:00"},{"id":"update_statuspage_circleci_rp9y8kszn25j_0kj0yscdj17h","status":"identified","body":"What's impacted\nCustomers running Docker jobs. Affected jobs are queuing rather than starting, resulting in elevated wait times.\n\nWhat can you expect\nDocker jobs may remain stuck in \"preparing environment\" for an extended period. Our engineers have identified the cause and are actively working to restore normal job startup times. Thank you for your patience while our team deploys a fix.\n\nNext update\nWe will provide another update as soon as more information is available","createdAt":"2026-06-23T19:04:41.685+00:00"},{"id":"update_statuspage_circleci_rp9y8kszn25j_v0fzqvgxf7hz","status":"monitoring","body":"What's happening\nOur engineers have deployed a fix and we are observing the job backlog clearing. Docker jobs are now starting normally.\n\nWhat's impacted\nCustomers who submitted Docker jobs during the incident window may still see some jobs completing with elevated queue times. Newly submitted jobs should be starting at normal times.\n\nWhat can you expect\nDocker job startup times are returning to normal. Thank you for your patience while our engineers resolved this issue.\n\nWe are monitoring the recovery and will provide a further update if the situation changes.","createdAt":"2026-06-23T19:29:42.422+00:00"},{"id":"update_statuspage_circleci_rp9y8kszn25j_pqz83s6jrdxz","status":"resolved","body":"The issue where customers running Docker jobs experienced significant delays, with jobs stuck in a \"preparing environment\" state, has now been resolved. We thank you for your patience while our team worked on implementing a fix.","createdAt":"2026-06-23T19:58:24.368+00:00"},{"id":"update_statuspage_circleci_rp9y8kszn25j_qtxrk80r06zj","status":"resolved","body":"## Summary\n\nOn Tuesday, June 23, 2026, our engineering team identified that a single Docker job cluster was operating below expected performance levels. Our infrastructure is organized into routing groups, each responsible for handling a specific segment of traffic, and this cluster was the sole member of its group. To address the performance issue, they initiated a standard two-step procedure to safely take it out of rotation: first, update the routing configuration to stop directing traffic to the cluster, then take it offline.\n\nWhen the team moved to take the cluster offline, the first step of the procedure was skipped. The cluster was removed from the available pool, but the routing configuration was not updated, meaning traffic continued to be directed to the routing group even though it had no capacity to process work. With no other clusters in the routing group to absorb the traffic, affected jobs had nowhere to go and began queuing rather than starting. This is a routine and safe operation when the two-step procedure is followed correctly, this incident exposed a gap in our tooling that allowed the procedure to be completed out of order.\n\nThe issue began at approximately 17:25 UTC and was fully resolved by 19:46 UTC, for a total impact window of roughly 2 hours and 20 minutes. Only Docker jobs were affected; other job types, including machine and macOS jobs, continued to operate normally.\n\nBeyond the missed step, several factors extended the impact window. An automated alert fired within a minute of the change deploying, but was dismissed as expected behavior. This delayed detection by nearly an hour. When engineers attempted to accelerate recovery by restoring a prior known-good version, they encountered unexpected issues with the rollback process, adding further delay before the fix could be deployed.\n\nThe original status page for this incident can be found [here](https://status.circleci.com/incidents/rp9y8kszn25j).\n\n## How Job Routing Works\n\nCircleCI uses Nomad as its job scheduling system for Docker-based jobs. When a job is triggered, it is routed to one of several Nomad clusters, organized into routing groups each responsible for handling a specific segment of traffic. This multi-cluster architecture is intentional: clusters are isolated from one another so that an issue in one does not impact the others, limiting blast radius and improving overall resiliency. Under normal circumstances, if a cluster becomes unavailable, traffic is automatically rerouted to the remaining operational clusters in its routing group, allowing jobs to continue without interruption.\n\n## What Happened\n\n\\(All times UTC\\)\n\nPrior to the incident, our engineering team identified that a single Docker job cluster was operating below expected performance levels. Our infrastructure is organized into routing groups, each responsible for handling a specific segment of traffic, and this cluster was the sole member of its group. To address the performance issue, they initiated a standard two-step procedure to safely take it out of rotation.\n\nAt 16:58, our engineering team applied a configuration change to remove the affected cluster from service. The first step of the two-step procedure was missed. The cluster was removed from the available pool, but the routing configuration was not updated, leaving traffic still being directed to a routing group with no capacity to process it.\n\nWhen the configuration change was fully deployed at 17:25, the affected jobs began to queue.\n\nAt 17:26, our monitoring alerted us to reduced job throughput on the affected cluster. Because throughput on that cluster was expected to drop as part of the cluster rotation procedure, the alert was interpreted as expected behavior and no action was taken.\n\nBy 18:19, a series of customer support tickets made clear that something beyond expected behavior was occurring. A formal incident was declared at 18:28 and engineers began investigating.\n\nBy 18:33, the team confirmed the impact was isolated to a subset of Docker jobs and identified a large backlog of queued jobs. At 18:41, a fix was merged to restore the affected cluster back into the available pool. Deployment of that fix was delayed because the CI smoke tests assume functioning clusters, leading to some rework of the pipeline itself. In parallel, the team attempted to accelerate recovery at 18:54 by restoring a prior known-good configuration directly, but encountered a conflict that left multiple active versions running simultaneously, preventing a clean rollback. The fix was fully deployed at 19:20 and Docker jobs began processing again.\n\nBy 19:28, the team confirmed the backlog was actively clearing and newly submitted jobs were starting at normal times.\n\nCustomers who had accumulated queued jobs during the incident period may have experienced a surge in concurrent job starts upon recovery, potentially hitting plan concurrency limits, which would have resulted in jobs still queueing while we processed the backlog.\n\nIn total, the period during which Docker jobs were unable to start lasted from approximately 17:25 to 19:20, roughly 1 hour and 55 minutes. The additional time from 19:20 to 19:46 was spent processing the accumulated backlog, during which some previously queued jobs completed with elevated wait times.\n\n## Future Prevention and Process Improvement\n\nWe are taking the following steps to prevent a recurrence and improve our response time:\n\n**We are strengthening our failover architecture.** Our multi-cluster design is built for resiliency, and while automatic failover mechanisms are already in place, this incident exposed a gap in our safeguards. We are closing that gap to ensure traffic is rerouted to healthy clusters across routing groups regardless of whether a cluster goes down unexpectedly or is intentionally taken offline.\n\n**We are enhancing our cluster rotation procedure.** This incident was possible because our tooling allowed an invalid state to exist, a cluster removed from the available pool while the routing configuration still directed traffic to it. We are updating this tooling to enforce that routing configuration and cluster availability are always updated together, and that the procedure cannot be completed out of order.\n\n**We are improving our monitoring and alerting.** The automated alert that fired during this incident was dismissed because it appeared consistent with the operation being performed. We are improving our monitoring so that unexpected side effects are surfaced clearly and are not confused with expected behavior.\n\n**We are improving our rollback capabilities.** During the incident, our team encountered delays when attempting to deploy the fix. We are auditing our deployment configurations to ensure that rollbacks can be executed more quickly and reliably during incidents.\n\nReliability is our top priority, and we are committed to ensuring that infrastructure upgrades like this one are executed with the appropriate safeguards in place.","createdAt":"2026-06-27T00:15:49.533+00:00"}],"access":{"plan":"public","keyed":false}}}