{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-07-24T10:18:42.113Z"
  },
  "data": {
    "id": "incident_statuspage_buildkite_8nd3cm8kqlnn",
    "slug": "buildkite-buildkite-service-disruption-2026-05-01",
    "title": "Buildkite service disruption",
    "summary": "Buildkite service disruption",
    "status": "resolved",
    "severity": "major",
    "startedAt": "2026-05-01T02:41:01.98+00:00",
    "updatedAt": "2026-05-08T06:09:24.413+00:00",
    "resolvedAt": "2026-05-01T03:30:52.534+00:00",
    "provider": {
      "slug": "buildkite",
      "name": "Buildkite"
    },
    "affectedServices": [],
    "links": {
      "html": "/incidents/buildkite-buildkite-service-disruption-2026-05-01",
      "api": "/api/v1/incidents/buildkite-buildkite-service-disruption-2026-05-01",
      "providerHtml": "/providers/buildkite"
    },
    "impactSummary": "Buildkite reported a major event for the affected tracked services.",
    "source": {
      "id": "source_buildkite_status",
      "kind": "official_status_page",
      "name": "Buildkite Status",
      "checkedAt": "2026-07-23T12:00:00Z",
      "officialUrl": "https://www.buildkitestatus.com",
      "statusPageUrl": "https://www.buildkitestatus.com"
    },
    "updates": [
      {
        "id": "update_statuspage_buildkite_8nd3cm8kqlnn_8c856w33mss6",
        "status": "identified",
        "body": "We've identified a service change that is causing a service disruption. We are reverting this change.",
        "createdAt": "2026-05-01T02:41:02.045+00:00"
      },
      {
        "id": "update_statuspage_buildkite_8nd3cm8kqlnn_f4t31vjzjpj9",
        "status": "monitoring",
        "body": "We've corrected the issue that caused this disruption and normal service has been restored. We are monitoring the situation now.",
        "createdAt": "2026-05-01T02:52:26.814+00:00"
      },
      {
        "id": "update_statuspage_buildkite_8nd3cm8kqlnn_czrrx3yrvd12",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2026-05-01T03:30:52.534+00:00"
      },
      {
        "id": "update_statuspage_buildkite_8nd3cm8kqlnn_syqfslrmc99b",
        "status": "resolved",
        "body": "### Service Impact\n\nBetween 02:29 and 02:48 UTC on May 1st 2026, the Buildkite dashboard, Agent API, REST API and GraphQL were severely degraded for the majority of customers.\n\nDuring this period, customers were unable to reliably load pages in the dashboard, and agents experienced errors when communicating with the Agent API.\n\nService was restored for all customers by 02:48 UTC, with a small number of follow-up support cases handled in the hours afterward.\n\n### Incident Summary\n\nAt 02:29 UTC, a database migration was applied that renamed a column on the users table in our foundation database, which is a core, unsharded table referenced across the Buildkite platform.\n\nThe migration ran before the new application code was deployed. During the rollout window, the application code already running in production still expected the original column name. Because database migrations run before application code is deployed, queries that loaded a user record began failing immediately across the product, and continued to fail until the column was manually restored.\n\nThe impact was detected at 02:29 UTC by an automated alert, and several customers reached out to support within minutes.\n\nInitial investigation focused on preparing an emergency rollback, but at approximately 02:41 UTC it was identified that the change was an unsafe migration and that a rollback would not restore service.\n\nWe then moved to manually recreate the renamed column on the users table. This change was applied at approximately 02:48 UTC, at which point error rates returned to normal levels across the affected services.\n\nTotal customer-facing impact was 19 minutes.\n\n### What contributed to this happening\n\nThree things came together to allow this change to ship without the safety pre-steps that would normally precede a destructive migration:\n\nThe migration was generated by an LLM-assisted workflow, which added the `safety_assured` annotation as part of the generated change. `safety_assured` is intended as an explicit engineer acknowledgement that the safety implications of a destructive migration have been considered and prepared for not as a routine flag added by tooling.\n\nWe use Danger \\([https://danger.systems/ruby/](https://danger.systems/ruby/)\\) to call out risky database migrations with specific guidance for certain operations such as index changes — but there was no specific guidance for `rename_column`. The generic migration warning that _did_ exist had, after dozens of routine migrations, become background noise.\n\nThe migration was bundled into the same PR as the application code that depended on the renamed column. This made the schema change easier to overlook in review, and meant the test suite for the PR only ever exercised the new code against the new schema, never the previously-deployed code against the new schema.\n\nThis combination meant the unsafe sequence reached production with no automated test or human review specifically focused on whether the _currently-running_ application could tolerate the schema change.\n\n### Changes we're making\n\nThis incident was caused by a database migration that was not adequately prepared for safe deployment, and exposed gaps in both our review process and our use of code generation tooling. We're making the following improvements to prevent similar incidents:\n\nDatabase migrations that are unsafe to apply against the currently-running application code will be flagged and blocked before they can be merged. We are still working through the exact mechanism for this \\(one option under active discussion is requiring schema migrations to ship in separate pull requests from any application code that depends on them\\), but the intent is the same: the test suite on a migration's PR exercises the post-migration schema against the application code already running in production — the exact intermediate state that exists between the migration running and any new code shipping — so that backwards-incompatibility shows up as a failing test on the migration PR itself, where it can be caught by branch protection rather than by a human reading a routine review comment.\n\nWe're extending Danger, the automated review tool that comments on every pull request that touches a database migration, to specifically highlight \\`rename\\_column\\` amongst the other higher-risk operations it already highlights, so that they stand out from routine migration changes during code review.\n\nWe're updating our internal engineering guidance \\([AGENTS.md](http://agents.md/)\\) to make clear that protective constructs such as safety\\_assured \\(from [https://github.com/ankane/strong\\_migrations](https://github.com/ankane/strong_migrations)\\) are intended as engineer-acknowledged sign-offs and should not be added by code-generation tooling on an engineer's behalf.\n\nFinally, we're reviewing how we surface and document the required pre-steps \\(such as adding ignored\\_columns\\) for breaking schema changes, so that the safe rollout sequence is harder to skip, particularly for changes to unsharded core tables where there is no opportunity for a phased rollout.\n\nThe changes described above protect against the _test-detectable_ class of schema/code mismatches. There remains a class of failure where a destructive migration to an unsharded core table affects every customer at once with no opportunity for a staged rollout. Our longer-term work on **cell-based deployments,** splitting customer data across multiple foundation databases, is what will give us the ability to roll out schema changes incrementally and detect issues against a small subset of customers before they reach the full fleet.",
        "createdAt": "2026-05-08T06:08:42.173+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}