{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Teams, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-10-06T19:35:18.874Z"
  },
  "data": {
    "id": "incident_statuspage_harness_fy36fxq22bw6",
    "slug": "harness-pipeline-executions-are-running-slow-in-prod1-2-3-2026-08-20",
    "title": "All modules are running slow in Prod1/2/3/4 due to cloud provider incident",
    "summary": "All modules are running slow in Prod1/2/3/4 due to cloud provider incident",
    "status": "resolved",
    "stale": false,
    "severity": "major",
    "startedAt": "2026-08-20T15:37:35.998+00:00",
    "updatedAt": "2026-08-26T16:20:43.14+00:00",
    "resolvedAt": "2026-08-20T19:23:42.742+00:00",
    "provider": {
      "slug": "harness",
      "name": "Harness"
    },
    "affectedServices": [
      {
        "slug": "harness-platform",
        "name": "Platform & dashboards"
      }
    ],
    "links": {
      "html": "/incidents/harness-pipeline-executions-are-running-slow-in-prod1-2-3-2026-08-20",
      "api": "/api/v1/incidents/harness-pipeline-executions-are-running-slow-in-prod1-2-3-2026-08-20",
      "providerHtml": "/providers/harness",
      "alerts": "https://outagedeck.com/account?stack=harness&utm_source=api&utm_medium=response&utm_campaign=api_alerts&utm_content=incident"
    },
    "impactSummary": "Harness reported a major event for the affected tracked services.",
    "source": {
      "id": "source_harness_status",
      "kind": "official_api",
      "name": "Harness Status",
      "checkedAt": "2026-10-06T19:30:11.451+00:00",
      "stale": false,
      "officialUrl": "https://status.harness.io",
      "statusPageUrl": "https://status.harness.io"
    },
    "updates": [
      {
        "id": "update_statuspage_harness_fy36fxq22bw6_qf9182pbv6g7",
        "status": "investigating",
        "body": "We are currently investigating this issue.",
        "createdAt": "2026-08-20T15:37:36.113+00:00"
      },
      {
        "id": "update_statuspage_harness_fy36fxq22bw6_ctsrj0k9xpqh",
        "status": "investigating",
        "body": "The slowness could cause either of below symptoms:\n- Pipelines not starting\n- Delays in execution\n- Pipelines being cancelled due to timeouts",
        "createdAt": "2026-08-20T16:05:28.137+00:00"
      },
      {
        "id": "update_statuspage_harness_fy36fxq22bw6_hs31k8r969cv",
        "status": "investigating",
        "body": "Our cloud provider is facing an active incident and we are following up.",
        "createdAt": "2026-08-20T16:22:40.263+00:00"
      },
      {
        "id": "update_statuspage_harness_fy36fxq22bw6_4fl141p9w6dg",
        "status": "investigating",
        "body": "We are continuing to investigate this issue.",
        "createdAt": "2026-08-20T16:32:42.087+00:00"
      },
      {
        "id": "update_statuspage_harness_fy36fxq22bw6_1n0rlfkxqd3f",
        "status": "identified",
        "body": "Our cloud provider has confirmed an ongoing incident impacting multiple regions. Harness pipelines have not experienced failures as a result, though some users may continue to experience slowness. We are monitoring the situation closely and will provide updates as more information becomes available.",
        "createdAt": "2026-08-20T16:50:35.407+00:00"
      },
      {
        "id": "update_statuspage_harness_fy36fxq22bw6_rzwnpcp7xqdc",
        "status": "monitoring",
        "body": "We are observing improved latencies across the board following the fix implemented by our cloud provider. We are continuing to monitor the situation closely and will provide further updates as warranted.\n\nWe noted some stuck executions for CI for couple of customers, which we are investigating",
        "createdAt": "2026-08-20T17:17:02.02+00:00"
      },
      {
        "id": "update_statuspage_harness_fy36fxq22bw6_fqydpm979sq2",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2026-08-20T19:23:42.742+00:00"
      },
      {
        "id": "update_statuspage_harness_fy36fxq22bw6_nv3w9xfl1hzq",
        "status": "resolved",
        "body": "# Summary\n\nOn 20 August 2026, beginning at approximately 15:00 UTC, the Harness platform experienced widespread performance degradation across all production environments. Pipeline executions that normally complete in around two minutes took seven to ten minutes. Continuous Delivery, Continuous Integration, pipeline orchestration, and Feature Management & Experimentation were all affected.\n\nGoogle Cloud Platform experienced a multi-product incident in the us-west1 region affecting Bigtable, Compute Engine, Google Kubernetes Engine, and persistent-disk I/O. Harness production infrastructure runs on persistent disks in that region. The degradation raised database operation latency from approximately 2 ms to over 10 ms at the 95th percentile, which in turn caused message-queue processing lag and propagated to every service that depends on timely database access.\n\n‌\n\n# Impact\n\nThis was a degradation, not an outage. Pipelines continued to execute and complete successfully throughout; they were slow rather than failing. No data was lost, and no customer work was dropped as a result of this incident.\n\n# **Root cause**\n\nHarness production infrastructure in the affected environments runs on Google Cloud Platform persistent disks in the us-west1 region. When that storage layer degraded, the effect propagated through the platform in a predictable chain:\n\n**Persistent-disk I/O degradation in us-west1.** Google Cloud Platform experienced a multi-product incident affecting Bigtable, Compute Engine, Google Kubernetes Engine, and persistent-disk performance. This was an infrastructure failure in the provider’s environment, outside Harness’s control.\n\n# **Preventive actions**\n\nAlthough Harness cannot prevent a cloud provider infrastructure failure. The actions below are aimed at detecting one faster and being better positioned to act on it.\n\n| **Action** |\n| --- |\n| Continue routine pre-testing of targeted cross-region database failovers, as performed during this incident, to keep failover readiness verified rather than assumed |\n| Assess full-stack multi-region failover readiness for future scenarios in which cross-region latency would be unacceptable |",
        "createdAt": "2026-08-26T04:57:12.107+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}