{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-07-24T10:19:20.617Z"
  },
  "data": {
    "id": "incident_statuspage_harness_pxl6nhjdnlwj",
    "slug": "harness-pipeline-updates-in-prod4-is-taking-time-2026-05-14",
    "title": "Pipeline Updates in Prod4 is taking time.",
    "summary": "Pipeline Updates in Prod4 is taking time.",
    "status": "resolved",
    "severity": "minor",
    "startedAt": "2026-05-14T13:36:21.554+00:00",
    "updatedAt": "2026-05-27T23:50:51.474+00:00",
    "resolvedAt": "2026-05-14T14:10:52.072+00:00",
    "provider": {
      "slug": "harness",
      "name": "Harness"
    },
    "affectedServices": [
      {
        "slug": "harness-cicd",
        "name": "CI/CD"
      },
      {
        "slug": "harness-platform",
        "name": "Platform & dashboards"
      }
    ],
    "links": {
      "html": "/incidents/harness-pipeline-updates-in-prod4-is-taking-time-2026-05-14",
      "api": "/api/v1/incidents/harness-pipeline-updates-in-prod4-is-taking-time-2026-05-14",
      "providerHtml": "/providers/harness"
    },
    "impactSummary": "Harness reported a minor event for the affected tracked services.",
    "source": {
      "id": "source_harness_status",
      "kind": "official_status_page",
      "name": "Harness Status",
      "checkedAt": "2026-07-23T12:00:00Z",
      "officialUrl": "https://status.harness.io",
      "statusPageUrl": "https://status.harness.io"
    },
    "updates": [
      {
        "id": "update_statuspage_harness_pxl6nhjdnlwj_hl8b7zvxzc8h",
        "status": "investigating",
        "body": "We are currently investigating this issue.",
        "createdAt": "2026-05-14T13:36:21.619+00:00"
      },
      {
        "id": "update_statuspage_harness_pxl6nhjdnlwj_r6psdgshvh26",
        "status": "investigating",
        "body": "We are continuing to investigate this issue.",
        "createdAt": "2026-05-14T13:39:49.592+00:00"
      },
      {
        "id": "update_statuspage_harness_pxl6nhjdnlwj_c35w64tlkbr5",
        "status": "investigating",
        "body": "We are continuing to investigate this issue.",
        "createdAt": "2026-05-14T13:40:21.939+00:00"
      },
      {
        "id": "update_statuspage_harness_pxl6nhjdnlwj_jjwq6pg2l449",
        "status": "identified",
        "body": "The issue has been identified and a fix is being implemented.",
        "createdAt": "2026-05-14T14:03:00.164+00:00"
      },
      {
        "id": "update_statuspage_harness_pxl6nhjdnlwj_xr51l68dx8xv",
        "status": "monitoring",
        "body": "A fix has been implemented and we are monitoring the results.",
        "createdAt": "2026-05-14T14:08:42.942+00:00"
      },
      {
        "id": "update_statuspage_harness_pxl6nhjdnlwj_tp8kz4j8bjn2",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2026-05-14T14:10:52.072+00:00"
      },
      {
        "id": "update_statuspage_harness_pxl6nhjdnlwj_gvn3s6r5vh3c",
        "status": "resolved",
        "body": "On May 14, 2026 , some customers running pipelines in the Prod4 production environment observed pipeline create and update requests that were slow or failed, and pipeline stages that did not start, produced no logs, and were eventually auto-aborted as “stuck”. \n\nThe issue was caused by an underlying compute node in our Prod4 cluster being recycled abruptly.\n\n## **Impact**\n\nDuring the incident window \\(approximately 5:38 PM PDT on May 14 to 9:47 PM PDT on May 14, 2026\\):\n\n* Some pipeline create and update requests on Prod4 were slow or failed.\n* Some Prod4 pipeline executions hung at the stage-start step, producing no logs, and were eventually auto-aborted as “stuck” after a timeout.\n* Behavior was intermittent — only pipelines whose requests were routed to an affected service pod were impacted; other pipelines continued to execute normally.\n\nThere was **no data loss**. The majority of pipelines on Prod4 continued to execute successfully throughout the incident — the primary impact was that affected create/update requests slowed down or failed, and a subset of pipelines could not progress and had to be aborted and re-run after mitigation. Overall service availability was degraded during this window.\n\n## **Root Cause**\n\nDuring the incident, an underlying compute node in our Prod4 cluster was recycled by the cloud provider without completing its normal graceful-drain process, so the supporting-service pods running on that node were terminated abruptly. As a result, in-flight requests from the backend service to those pods were left without a response. \n\n## **Mitigation**\n\nHarness completed the following immediate mitigation steps:\n\n* Restarted the affected supporting-service pods to restore healthy targets.\n* Restarted the pipeline service in Prod4 to clear the blocked worker threads. This is what fully restored normal pipeline create/update behavior and stage-start behavior; restarting only the supporting service was not enough on its own.\n* Confirmed pipeline executions returned to normal and updated the status page to mitigated.\n\nThese actions restored pipeline execution behavior and resolved the customer-facing impact.\n\n## **Action Items**\n\nTo reduce the risk of recurrence and improve detection, the following actions are in various stages of being implemented:\n\n* Optimize timeouts to the pipeline service’s plan-creation requests so that when a supporting service goes away unexpectedly, the worker threads recover automatically instead of remaining blocked.\n* Investigate the abrupt node-recycle behavior in Prod4 with our cloud provider to ensure pods running on a recycled node receive a graceful shutdown signal in the future.\n* Add proactive paging alerts on service worker-thread saturation, so this failure mode is detected before it becomes a impacting issue",
        "createdAt": "2026-05-27T23:47:55.197+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}