{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-07-24T10:19:54.929Z"
  },
  "data": {
    "id": "incident_statuspage_buildkite_j8qlchvxr102",
    "slug": "buildkite-jobs-not-starting-on-hosted-agents-and-agent-stack-k8s-2026-05-07",
    "title": "Jobs not starting on hosted agents and agent-stack-k8s",
    "summary": "Jobs not starting on hosted agents and agent-stack-k8s",
    "status": "resolved",
    "severity": "major",
    "startedAt": "2026-05-07T08:48:08.296+00:00",
    "updatedAt": "2026-05-15T02:01:23.443+00:00",
    "resolvedAt": "2026-05-07T09:37:50.303+00:00",
    "provider": {
      "slug": "buildkite",
      "name": "Buildkite"
    },
    "affectedServices": [
      {
        "slug": "buildkite-pipelines",
        "name": "Pipelines & job queue"
      }
    ],
    "links": {
      "html": "/incidents/buildkite-jobs-not-starting-on-hosted-agents-and-agent-stack-k8s-2026-05-07",
      "api": "/api/v1/incidents/buildkite-jobs-not-starting-on-hosted-agents-and-agent-stack-k8s-2026-05-07",
      "providerHtml": "/providers/buildkite"
    },
    "impactSummary": "Buildkite reported a major event for the affected tracked services.",
    "source": {
      "id": "source_buildkite_status",
      "kind": "official_status_page",
      "name": "Buildkite Status",
      "checkedAt": "2026-07-23T12:00:00Z",
      "officialUrl": "https://www.buildkitestatus.com",
      "statusPageUrl": "https://www.buildkitestatus.com"
    },
    "updates": [
      {
        "id": "update_statuspage_buildkite_j8qlchvxr102_5p1plrn1s16s",
        "status": "investigating",
        "body": "We've spotted that something has gone wrong. We're currently investigating the issue with new builds not starting.",
        "createdAt": "2026-05-07T08:48:08.39+00:00"
      },
      {
        "id": "update_statuspage_buildkite_j8qlchvxr102_527n91p2p09x",
        "status": "investigating",
        "body": "We've identified issue with job acquiring endpoint. We're rolling back now. We'll provide next update in ~20 minutes.",
        "createdAt": "2026-05-07T09:09:01.423+00:00"
      },
      {
        "id": "update_statuspage_buildkite_j8qlchvxr102_tyfb40xx3r9q",
        "status": "identified",
        "body": "We're currently seeing recovery at 50% rate. We'll provide next update soon.",
        "createdAt": "2026-05-07T09:24:55.943+00:00"
      },
      {
        "id": "update_statuspage_buildkite_j8qlchvxr102_kxcyvlbklhkz",
        "status": "resolved",
        "body": "We've reverted a change that caused stale environment variables provided to acquire job used in Hosted Agents, agent-stack-k8s and other agent implementations using acquire job.",
        "createdAt": "2026-05-07T09:37:50.303+00:00"
      },
      {
        "id": "update_statuspage_buildkite_j8qlchvxr102_6cv9bf0lkh9y",
        "status": "resolved",
        "body": "## Service Impact\n\nOn May 7th between 08:14 and 09:25 UTC customers using [hosted agents](https://buildkite.com/docs/agent/buildkite-hosted), the [k8s stack](https://buildkite.com/docs/agent/self-hosted/agent-stack-k8s) and the Buildkite agent [acquire job feature](https://buildkite.com/docs/agent/cli/reference/start#run-a-single-job) experienced failures when starting a job, resulting in an error message `Missing agent. See: buildkite-agent bootstrap --help`.\n\n## Incident Summary\n\nAs part of our efforts to improve the performance of our platform we shipped a change to how database commits were grouped together. This change inadvertently caused certain job environment variables to be omitted when a job was assigned via the acquire method. These variables are key to our integration with hosted agents as well as customers using our Kubernetes stack and their omission caused any jobs launched via this method to fail.\n\nWhile we had test coverage that ensured these variables were populated, these tests did not exercise the code path used by acquire job effectively enough to indicate this problem before it was deployed. Furthermore, while parts of the change were placed behind a feature flag, the refactor of the code that caused this bug were not.\n\nAttempts to restore service by rolling back this change were hindered by the current revision being selected for deploy instead of the previous. The initial rollback was triggered at around 08:40 UTC, but it wasn’t until 09:05 we realised the mistake and began the rollback to the correct revision. The rollback started deploying at 09:16 and at 09:25 service was fully restored.\n\n‌\n\n![](https://lh3.googleusercontent.com/d/1VjNtAjf5IiIFcG7iYSH-9xHWNc5KCu-n)\n## Changes we're making\n\nWe have added additional rollback gates making it easier to identify when the incorrect revisions has been selected.\n\nOur test suite will be expanded to include contract tests for APIs used by Hosted Agent and the Kubernetes stack.\n\nAdditionally, we are configuring Hosted Agent synthetic tests to automatically page on-call engineers when failures occur, improving our response times.",
        "createdAt": "2026-05-15T01:15:46.81+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}