{"meta":{"version":"v1","pricing":{"public":{"label":"Public","description":"Read-only API access for lightweight status checks and public integrations."},"premium":{"label":"Premium (early access)","description":"API keys with higher hourly quotas are available now; webhooks, alerting, and exports are on the roadmap."}},"generatedAt":"2026-07-09T05:24:52.815Z"},"data":{"id":"incident_statuspage_circleci_kvzp8x242n34","slug":"circleci-jobs-stuck-in-running-state-2025-12-03","title":"Jobs stuck in running state","summary":"Jobs stuck in running state","status":"resolved","severity":"minor","startedAt":"2025-12-03T16:53:11+00:00","updatedAt":"2025-12-18T21:30:13.274+00:00","resolvedAt":"2025-12-03T18:12:05.619+00:00","provider":{"slug":"circleci","name":"CircleCI"},"affectedServices":[{"slug":"circleci-pipelines","name":"Pipelines & Workflows"}],"links":{"html":"/incidents/circleci-jobs-stuck-in-running-state-2025-12-03","api":"/api/v1/incidents/circleci-jobs-stuck-in-running-state-2025-12-03","providerHtml":"/providers/circleci"},"impactSummary":"CircleCI reported a minor event for the affected tracked services.","source":{"id":"source_circleci_status","kind":"official_api","name":"CircleCI Status","checkedAt":"2026-07-09T05:20:23.992+00:00","officialUrl":"https://status.circleci.com","statusPageUrl":"https://status.circleci.com"},"updates":[{"id":"update_statuspage_circleci_kvzp8x242n34_hcmp1rmfcmn1","status":"investigating","body":"At 16:20 UTC, we began experiencing delays in job triggering and starts across all resource classes. Some workflows and jobs may appear stuck in a running state.\n\nWhat’s impacted: Job triggering is experiencing delays or is stuck. This affects all resource classes and executors.\n\nWhat to expect: If you have workflows that appear stuck and haven’t started, we recommend manually rerunning them.\n\nWe are actively investigating the root cause and working to restore normal processing speeds.  Next update: We will provide an update within 30 minutes or earlier with our progress","createdAt":"2025-12-03T16:53:11.159+00:00"},{"id":"update_statuspage_circleci_kvzp8x242n34_9m1yw50lx29n","status":"monitoring","body":"As of 16:32 UTC, job triggering and workflow starts have returned to normal operation across all resource classes. The impact was limited to a 12-minute window between 16:20 and 16:32 UTC.\n\nWhat's impacted:  All new jobs and workflows are now starting normally.\n\nWhat to expect: If you have workflows or jobs that were stuck during the 16:20-16:32 UTC window, please manually rerun them.\n\nWe are continuing to investigate the root cause of this disruption and will provide an update within 30 minutes or once our investigation is complete.","createdAt":"2025-12-03T17:27:55.749+00:00"},{"id":"update_statuspage_circleci_kvzp8x242n34_pq5t87pjkcgv","status":"resolved","body":"Between 16:20 and 16:32 UTC, job triggering and workflow starts experienced disruptions across all resource classes due to memory pressure on our internal job distributor systems. We identified the issue and scaled our infrastructure to handle the load. Services returned to normal operation at 16:32 UTC.\n\nWhat was impacted: Job triggering and workflow starts were disrupted for 12 minutes. Some workflows and jobs appeared stuck in a running state during this window.\n\nResolution: Our systems are now operating normally with additional capacity in place to prevent similar disruptions. If you had workflows or jobs that were stuck during this window, please manually rerun them.\n\nThe incident is now resolved and we will be conducting a thorough review to understand what triggered the memory pressure and identify any additional preventive measures","createdAt":"2025-12-03T18:12:05.619+00:00"},{"id":"update_statuspage_circleci_kvzp8x242n34_n3v5fgv9jtf0","status":"resolved","body":"## Summary\n\n_All timestamps are in UTC_\n\nOn December 2, 2025, at 16:20, we deployed a code change to workflow status computation that introduced a race condition in how workflows are terminated. This caused the following impacts:\n\n**Duplicated auto-rerun workflows \\(Dec 2, 16:20 - Dec 3, 14:20\\):** Some customers using auto-rerun experienced duplicate workflows being created and duplicate email notifications.\n\n**API and UI degradation \\(Dec 3, 12:10 - 22:50\\):** Some customers experienced issues with the Workflows API and errors in the UI when trying to view pipelines.\n\n**Job execution delays \\(Dec 3, 16:20 - 16:32\\):** Several customers experienced delays in job execution, with some jobs failing to start or becoming stuck.\n\nThe incident was fully resolved on **December 3 at 22:50.**\n\nWe thank our customers for their patience and understanding as we worked to resolve this incident.\n\nThis incident tracked as multiple separate issues and status pages:\n\n* Pipelines page not loading \\([Link 1](https://status.circleci.com/incidents/21cysyp8d333), [Link 2](https://status.circleci.com/incidents/fjpg0vmh476k)\\)\n* Jobs stuck in running state \\([Link](https://status.circleci.com/incidents/kvzp8x242n34)\\)\n* Jobs not starting \\([Link](https://status.circleci.com/incidents/jq4bgq2sjt1r)\\)\n\n## What Happened\n\nOn December 2, 2025, at 16:20, we deployed a code change that introduced a race condition in how workflows are terminated. When workflows failed with concurrent jobs terminating, our system published duplicate workflow completion events. For some customers with auto-rerun enabled, these duplicate events triggered duplicate workflows to be created, causing cascading issues across our platform.\n\n_**Initial Detection and Response \\(Dec 2, 21:40 - Dec 3, 14:20\\)**_\n\nAt 21:40 on December 2nd, we received the first customer report of excessive workflow executions. Our notifications team was also alerted to a high number of notifications being sent. We initially thought it was an isolated issue related to customer automation, so we flagged the affected project at 21:46 and saw recovery on the notifications side.\n\nAt 21:51 we learned that a different customer reported duplicate workflow failure emails. We suspected it might be related to a recent feature rollout, but investigation did not identify a root cause. We initially thought the elevated rate of messages from the first customer somehow impacted other customers. By 22:19, no other customer had reported similar issues and metrics were back to normal, so we decided to continue the investigation the next day.\n\nAt 9:55 on December 3rd, we received alerts to elevated notifications and we resumed investigation. We escalated to a full incident at 13:44. We learned that the notifications issues seemed to be caused by auto-reruns, so we disabled auto-rerun system-wide at 13:56. We identified and rolled back the problematic code change at 14:15, then re-enabled auto-rerun at 14:20.\n\nThis resolved the primary issue of duplicate workflows and notifications being created.\n\n38 projects were impacted by the auto-rerun issue with a total of 33,135 duplicate workflows created. It also led to duplicate email notifications for 18 organizations.\n\nBecause the issue manifested across multiple domains \\(Orchestration, Notifications, API/UI, and Execution\\), we initially investigated these as separate incidents. It took time to correlate these incidents back to a single root cause. However, this fix did not immediately resolve all customer-facing issues due to downstream impacts.\n\n### **Downstream Impact**\n\nThe excessive workflows caused cascading issues across our platform:\n\n_**Notification Issues \\(Dec 2, 21:40 - Dec 3, 14:20\\)**_  \n18 organizations received duplicate email notifications for workflow failures.\n\n_**API and UI Issues \\(Dec 3, 12:10 - 22:50\\)**_  \nAt 11:54, our monitoring detected a service queue growing due to duplicate events. While investigating, we discovered the Pipelines page was struggling to load starting at 12:10. The issue was intermittent and was initially resolved at 15:11. However, at 15:30 API errors returned and we continued to investigate.\n\nAt 21:30 we identified that some pipelines had over 10,000 workflows, causing our APIs to slow down and fail. Affected customers experienced failed requests to the Workflows API and errors when trying to view pipelines. We deployed targeted fixes between 22:16 and 22:50 that filtered out problematic pipelines, restoring API performance.\n\n_**Job Execution Delays \\(Dec 3, 16:20 - 16:32\\)**_\n\nAt 16:18, our monitoring alerted us to issues with our job orchestration system. While investigating, we identified a drop in jobs being processed. At 16:20, our job execution infrastructure began experiencing memory pressure due to increased API traffic from the duplicate workflows. This memory pressure disrupted normal job processing and cancellation handling, and caused services to restart. Job processing recovered at 16:32. During this window, approximately 3,500 jobs were delayed, and 1,100 jobs \\(3% of tasks\\) failed to start. Additionally, 8,000 jobs became stuck in a canceling state. We then scaled our execution infrastructure to prevent further memory pressure.\n\nWe later identified that this execution impact was connected to the API issues described above, demonstrating the cascading nature of the incident across multiple systems.\n\n### **Resolution**\n\nAll systems fully recovered by 22:50 UTC on December 3rd. We have refunded credits consumed by duplicate workflows for all 38 impacted projects on December 5th.\n\n## Future Prevention and Process Improvement\n\nWe're taking several actions to prevent similar incidents and improve our detection and response:\n\n**Preventing Duplicate Workflows**\n\nWe're putting safeguards in place to prevent race conditions when workflows are terminated and we're also adding constraints to prevent duplicate auto-rerun workflows from being created in the first place.\n\n**Improved Monitoring and Alerting**\n\nWhile we have monitors in place, we didn't get paged for some of the issues, which delayed our incident response. As a result, we've updated existing monitoring alerts to page on-call engineers for execution infrastructure API errors and service restarts, and we're adding new alerts for UI-related issues.\n\n**Better System Resilience**\n\nWe will be updating the Pipelines page to handle large numbers of workflows more gracefully while also looking to understand and improve how our systems behave under pressure.\n\n**Incident Response**\n\nOne of the key challenges in this incident was the difficulty in correlating that these incidents all came from the same root cause. As a result we're taking the following actions:\n\n* Establishing better communication protocols to quickly correlate related incidents across teams\n* Improving coordination when mitigations in one system may impact downstream services","createdAt":"2025-12-18T21:30:10.571+00:00"}],"access":{"plan":"public","keyed":false}}}