{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-07-23T09:00:15.129Z"
  },
  "data": {
    "id": "incident_statuspage_harness_wvnqm2pt9j71",
    "slug": "harness-hosted-ci-builds-failing-intermittently-2026-05-27",
    "title": "Hosted CI Builds Failing Intermittently",
    "summary": "Hosted CI Builds Failing Intermittently",
    "status": "resolved",
    "severity": "major",
    "startedAt": "2026-05-27T12:35:32.364+00:00",
    "updatedAt": "2026-06-04T15:47:06.927+00:00",
    "resolvedAt": "2026-05-28T01:48:45.368+00:00",
    "provider": {
      "slug": "harness",
      "name": "Harness"
    },
    "affectedServices": [
      {
        "slug": "harness-cicd",
        "name": "CI/CD"
      }
    ],
    "links": {
      "html": "/incidents/harness-hosted-ci-builds-failing-intermittently-2026-05-27",
      "api": "/api/v1/incidents/harness-hosted-ci-builds-failing-intermittently-2026-05-27",
      "providerHtml": "/providers/harness"
    },
    "impactSummary": "Harness reported a major event for the affected tracked services.",
    "source": {
      "id": "source_harness_status",
      "kind": "official_api",
      "name": "Harness Status",
      "checkedAt": "2026-07-23T08:55:04.121+00:00",
      "officialUrl": "https://status.harness.io",
      "statusPageUrl": "https://status.harness.io"
    },
    "updates": [
      {
        "id": "update_statuspage_harness_wvnqm2pt9j71_sbxclmbzl29v",
        "status": "investigating",
        "body": "We are currently investigating this issue.",
        "createdAt": "2026-05-27T12:35:32.599+00:00"
      },
      {
        "id": "update_statuspage_harness_wvnqm2pt9j71_dz3jc8xh9b1b",
        "status": "identified",
        "body": "The issue has been identified and a fix is being implemented.",
        "createdAt": "2026-05-27T13:43:07.675+00:00"
      },
      {
        "id": "update_statuspage_harness_wvnqm2pt9j71_4gghvxgpx4nf",
        "status": "identified",
        "body": "We are continuing to work on a fix for this issue.",
        "createdAt": "2026-05-27T14:06:23.242+00:00"
      },
      {
        "id": "update_statuspage_harness_wvnqm2pt9j71_84gtkx63lc52",
        "status": "identified",
        "body": "Harness is continuing to investigate and implement changes to fully restore functionality.  At this time, we are still seeing some CI Builds intermittently fail.",
        "createdAt": "2026-05-27T15:20:54.3+00:00"
      },
      {
        "id": "update_statuspage_harness_wvnqm2pt9j71_3k3sc16pr2tc",
        "status": "identified",
        "body": "Harness has implemented a change and are seeing failures reduced, but are continuing to work on completing mitigation.  Customers can expect executions to succeed more frequently.\n\nProd1/Prod2 are back to normal.",
        "createdAt": "2026-05-27T16:32:48.342+00:00"
      },
      {
        "id": "update_statuspage_harness_wvnqm2pt9j71_mpp7md8wv7fg",
        "status": "identified",
        "body": "Harness is currently implementing a failover to mitigate the intermittent issue for some customers",
        "createdAt": "2026-05-27T17:54:26.821+00:00"
      },
      {
        "id": "update_statuspage_harness_wvnqm2pt9j71_fqyw4fztc4b3",
        "status": "monitoring",
        "body": "A fix has been implemented and we are monitoring the results.",
        "createdAt": "2026-05-27T23:53:11.755+00:00"
      },
      {
        "id": "update_statuspage_harness_wvnqm2pt9j71_v4cqlmq42zyl",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2026-05-28T01:48:45.368+00:00"
      },
      {
        "id": "update_statuspage_harness_wvnqm2pt9j71_fg764yv08l9c",
        "status": "resolved",
        "body": "### **Summary**\n\nBetween May 27 and June 1, 2026, some Harness CI customers experienced pipeline execution failures with the error:\n\n`failed to call LE.RetryStartStep: context deadline exceeded`\n\n###  **Impact**\n\nAffected customers saw intermittent CI pipeline failures during step execution.\n\n Existing pipeline definitions, customer data, source code, and artifacts were not impacted.\n\n### **Root Cause**\n\nThe root cause was a deadlock in the Light Engine logging path. When the log service returned an error, the Light Engine log writer attempted to reacquire a mutex it already held. This caused the Light Engine process to freeze, which led to step execution timeouts and pipeline failures.  \n\n### **Mitigation and Resolution**\n\nHarness Engineering took multiple mitigation steps during the incident, including:\n\n* Rolled back affected runner versions where needed\n* Increased relevant timeout configurations\n* Reduced log-service load and latency\n* Temporarily disabled the affected livelog streaming path\n* Migrated selected workloads across regions and infrastructure providers\n* Pinned a fixed Light Engine version through runner configuration\n\nThe final fix addressed the mutex deadlock in the Light Engine log writer and prevented the same lock from being reacquired while already held.  \n\n### **Prevention and Follow-Up Actions**\n\nHarness is taking the following actions to reduce recurrence risk:\n\n* Improve deadlock detection in critical concurrent code paths\n* Strengthen error handling for log-service interactions\n* Add better monitoring for Light Engine process health\n* Improve safeguards around logging-path failures\n* Continue reviewing runner rollout and validation processes",
        "createdAt": "2026-06-04T15:30:24.637+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}