Review active and historical incidents affecting GitLab.
The same incident history is on the public JSON API and the RSS feed, no account needed. Want it pushed instead of polled? Free email alerts, or Slack, Teams, Discord, and webhooks on a paid plan, fire the moment GitLab reports something new.
The incident affecting GitLab.com CI/CD jobs using instance runners is now resolved. Some jobs were delayed or remained pending while a manually initiated pg_repack database maintenance operation caused contention on the CI database, affecting runner communication and job pickup.
Performance remains stable and prevention steps are being implemented. Find more information here: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22807
Our upstream provider mitigated a genuine distributed HTTP DDoS attack against gitlab.com, but incorrectly blocked a small amount of legitimate traffic. This has now been resolved.
The fix has been implemented and the issue is now resolved. We have not observed any active user impact. More information: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22783
The deployment failures affecting the GitLab Agent for Kubernetes (KAS) have been resolved. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22777
The elevated 403 errors affecting GitLab.com have been resolved and error rates have returned to normal levels. More details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22762
The error rate remains within normal thresholds and the issue is now resolved. Further details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22729
All deletion queues have been processed for projects, we expect the same to continue for group deletions. This incident is moved to resolved. More information here: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22714
The queue of pending jobs on the affected runners has returned to normal and has stayed at that level. This incident is now resolved. Please open a ticket with support if you are still seeing job delays. Details: gitlab.com/gitlab-com/gl-infra/production/-/work_items/22702
We're seeing consistent background job and webhook delivery. The issue is now resolved.
We've applied mitigations that have stabilized the job queue. The issue is now resolved. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22654.
Webhook delivery failures have subsided in logs, and customer reports indicate no further errors. This incident is considered resolved.
Duo Code Review sessions on GitLab.com have returned to normal operation. This issue is now resolved.
After a period of monitoring we have seen continued Gitaly stability and consider the incident resolved. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22625.
We have continued to observe normal error rates. We will now be marking this incident as resolved. Details surrounding the incident can be seen here: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22579
Earlier login problems have been resolved and logins are working normally again. This incident is now considered closed. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22551
We've identified and mitigated the root cause of this issue. Service has remained stable throughout the monitoring period, and this incident is now resolved.
Mitigation efforts have been successfully completed, and service has returned to normal. This incident has been resolved.
We are seeing error rates return to normal. This issue has been resolved. Details at https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519
The fix has been completed in production. This incident is now resolve. More: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22470
Incident has now been mitigated. CI pipelines are now resuming. Engineers will continue to monitor this recovery.
Operations have returned to normal and this issue is resolved.
This issue is resolved and performance on GitLab.com has returned to normal.
Incident has now been mitigated and runner backlogs are now back to normal. CI pipelines are now resuming. Engineers will continue to monitor this recovery.
We have continued to observe successful SSH interactions after implementing our mitigating changes. We will now mark this as resolved. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22364