Incident with CodeQL, Webhooks, Notifications, and Slack Integration

We are investigating reports of degraded performance for CodeQL

CodeQL actions are currently experiencing delays, which may result in those actions being stuck in a pending state or having failed due to a timeout.

We're continuing to investigate issues with CodeQL actions workflows. We're additionally seeing delays for notifications, webhooks, and the Slack integration.

Webhooks is experiencing degraded performance. We are continuing to investigate.

We've established that most delays are related to a queuing service and are working to scale out. Early signals from the scale-out are showing signs of recovery for some services. We'll provide an update when services are fully recovered.

Webhooks is operating normally.

Webhooks have fully recovered. Continuing to work on recovery for the other services.

CodeQL has fully recovered. We're continuing to work on recovery for the remaining impacted services.

All services have fully recovered.

On May 12, 2026, between 13:41 and 17:43 UTC, some services experienced delays in processing. For the Code Scanning service, 53% of check runs took over 15 minutes to complete. Additionally, notifications took an average of 22 minutes to be delivered and Slack integration webhooks took an average of 20 minutes to be delivered. The delays were caused by replication lag due to an internal database migration, resulting in insufficient worker capacity for our high rate of job enqueues. <br /><br />We mitigated the impact by scaling our processing workers to handle the increased load. All services returned to normal processing times after the mitigation was applied. <br /><br />We are working to create dedicated worker pools for some of our high usage shared queues to help prevent this in the future.