{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Teams, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-11-07T17:40:39.749Z"
  },
  "data": {
    "id": "incident_statuspage_buildkite_t1h2d1f4y6qx",
    "slug": "buildkite-issues-with-webhook-ingestion-graphql-queries-2026-09-22",
    "title": "Issues with webhook ingestion, GraphQL queries",
    "summary": "Issues with webhook ingestion, GraphQL queries",
    "status": "resolved",
    "stale": false,
    "severity": "major",
    "startedAt": "2026-09-22T20:07:55.804+00:00",
    "updatedAt": "2026-09-28T04:24:38.763+00:00",
    "resolvedAt": "2026-09-22T20:50:02.597+00:00",
    "provider": {
      "slug": "buildkite",
      "name": "Buildkite"
    },
    "affectedServices": [],
    "links": {
      "html": "/incidents/buildkite-issues-with-webhook-ingestion-graphql-queries-2026-09-22",
      "api": "/api/v1/incidents/buildkite-issues-with-webhook-ingestion-graphql-queries-2026-09-22",
      "providerHtml": "/providers/buildkite",
      "alerts": "https://outagedeck.com/account?stack=buildkite&utm_source=api&utm_medium=response&utm_campaign=api_alerts&utm_content=incident"
    },
    "impactSummary": "Buildkite reported a major event for the affected tracked services.",
    "source": {
      "id": "source_buildkite_status",
      "kind": "official_api",
      "name": "Buildkite Status",
      "checkedAt": "2026-11-07T17:30:11.663+00:00",
      "stale": false,
      "officialUrl": "https://www.buildkitestatus.com",
      "statusPageUrl": "https://www.buildkitestatus.com"
    },
    "updates": [
      {
        "id": "update_statuspage_buildkite_t1h2d1f4y6qx_vn7y3y8zs8wm",
        "status": "investigating",
        "body": "We're seeing an issue affecting webhook ingestion and execution of GraphQL queries and are investigating the cause.",
        "createdAt": "2026-09-22T20:07:55.844+00:00"
      },
      {
        "id": "update_statuspage_buildkite_t1h2d1f4y6qx_bb9vr99zb63w",
        "status": "identified",
        "body": "Our engineers have addressed the issue and we're monitoring recovery.",
        "createdAt": "2026-09-22T20:26:22.873+00:00"
      },
      {
        "id": "update_statuspage_buildkite_t1h2d1f4y6qx_2nvt2z03109f",
        "status": "resolved",
        "body": "Services are fully recovered. Webhook ingestion was affected between 19:09 UTC - 19:32 UTC.",
        "createdAt": "2026-09-22T20:50:02.597+00:00"
      },
      {
        "id": "update_statuspage_buildkite_t1h2d1f4y6qx_6kgp2588bs1v",
        "status": "resolved",
        "body": "## Service Impact\n\nOn September 22, 2026, between 19:09 and 19:32 UTC, customers experienced failures ingesting webhooks and executing GraphQL queries. Some web application requests also failed or timed out.\n\nDuring this period, 2.0% of requests to our webhook ingestion endpoints and 20.3% of requests to our external GraphQL endpoint returned HTTP 500 errors.\n\nWebhook ingestion errors ceased by 19:30 UTC, and external GraphQL errors returned to background levels by 19:32 UTC.\n\n## Incident Summary\n\nWhile investigating a separate API performance issue, we made a manual configuration change intended to rollback a routing change for one database proxy. The command instead changed the routing configuration for another database proxy, disrupting access to a database shard used internally by Buildkite.\n\nSome customer-facing operations search across all database shards to locate their shard, where routing information isn't available in the request. Those requests round robin through available shards, and could encounter the unavailable shard and wait for a database connection to time out. This extended the impact beyond our internal workloads.\n\nWe reverted the incorrect configuration and verified that the database proxy had healthy endpoints and load-balancer targets.\n\nTwo contributing factors allowed the change to cause an outage: the command was not independently checked before execution, and our infrastructure accepted a Service configuration that selected the wrong database proxy workload. The cross-shard lookup behaviour then allowed the disruption to affect requests for other customers.\n\nThis configuration-change outage was separate from the API performance issue we were originally investigating.\n\n## Changes we're making\n\n* Safer operational procedures: We have updated our runbook and introduced an explicit second person review requirement for manual production commands.\n* Automated safeguards: We’ve added validation to prevent database proxy routing changes from selecting the wrong workload.\n* Improved failure isolation: We’ve been working on an ongoing initiative to make database shard discovery more resilient, eliminating the remaining cases where an unavailable shard may impact requests for other customers.",
        "createdAt": "2026-09-28T04:12:17.018+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}