Skip to content

Incident detail

Elevated Errors Affecting AI Gateway Health Checks

We are seeing error rates return to normal. This issue has been resolved. Details at https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

Resolved incidentMajor

Timeline window

to

Outage alerts

Get alerted the next time GitLab breaks

Free email alerts for up to 5 providers — no card, live in about a minute. Paid plans add Slack, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Investigating

    We're aware of elevated errors affecting AI Gateway health checks. We've identified a likely cause and are actively working on a fix. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  2. Identified

    We have identified the problematic change causing this behavior and are working to revert it. Details will be shared in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  3. Identified

    We're continuing to work on reverting the change causing this issue. Details will be shared in: gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  4. Identified

    The revert remains in progress. Additional updates will be shared in: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  5. Monitoring

    Deployment for the revert has reached production. Manual verification is successful with error rates trending down. We are continuing to monitor. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  6. Monitoring

    We are monitoring the situation to ensure that services have returned to normal. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519

  7. Resolved

    We are seeing error rates return to normal. This issue has been resolved. Details at https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22519