{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-07-23T09:49:55.338Z"
  },
  "data": {
    "id": "incident_statuspage_harness_v4nxhkv1mvdg",
    "slug": "harness-deployment-degradation-failures-in-prod1-2-3-2026-05-07",
    "title": "Deployment Degradation – Failures in Prod1,2,3",
    "summary": "Deployment Degradation – Failures in Prod1,2,3",
    "status": "resolved",
    "severity": "minor",
    "startedAt": "2026-05-07T07:54:42+00:00",
    "updatedAt": "2026-05-20T00:46:55.987+00:00",
    "resolvedAt": "2026-05-07T11:23:53.42+00:00",
    "provider": {
      "slug": "harness",
      "name": "Harness"
    },
    "affectedServices": [
      {
        "slug": "harness-cicd",
        "name": "CI/CD"
      },
      {
        "slug": "harness-feature-flags",
        "name": "Feature flags & FME"
      },
      {
        "slug": "harness-platform",
        "name": "Platform & dashboards"
      }
    ],
    "links": {
      "html": "/incidents/harness-deployment-degradation-failures-in-prod1-2-3-2026-05-07",
      "api": "/api/v1/incidents/harness-deployment-degradation-failures-in-prod1-2-3-2026-05-07",
      "providerHtml": "/providers/harness"
    },
    "impactSummary": "Harness reported a minor event for the affected tracked services.",
    "source": {
      "id": "source_harness_status",
      "kind": "official_api",
      "name": "Harness Status",
      "checkedAt": "2026-07-23T09:45:03.435+00:00",
      "officialUrl": "https://status.harness.io",
      "statusPageUrl": "https://status.harness.io"
    },
    "updates": [
      {
        "id": "update_statuspage_harness_v4nxhkv1mvdg_942rpv5jfvhg",
        "status": "investigating",
        "body": "Deployment Degradation – Failures in Prod1,2,3",
        "createdAt": "2026-05-07T07:54:42.769+00:00"
      },
      {
        "id": "update_statuspage_harness_v4nxhkv1mvdg_4s03ck5fdyfr",
        "status": "investigating",
        "body": "We are continuing to investigate this issue.",
        "createdAt": "2026-05-07T08:17:46.937+00:00"
      },
      {
        "id": "update_statuspage_harness_v4nxhkv1mvdg_3377dtsq45j8",
        "status": "investigating",
        "body": "We are continuing to investigate this issue.",
        "createdAt": "2026-05-07T08:43:12.848+00:00"
      },
      {
        "id": "update_statuspage_harness_v4nxhkv1mvdg_pr20vw2ffny2",
        "status": "monitoring",
        "body": "A fix has been implemented and we are monitoring the results.",
        "createdAt": "2026-05-07T09:05:32.885+00:00"
      },
      {
        "id": "update_statuspage_harness_v4nxhkv1mvdg_rc1381nbw282",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2026-05-07T11:23:53.42+00:00"
      },
      {
        "id": "update_statuspage_harness_v4nxhkv1mvdg_284y6b23zc34",
        "status": "resolved",
        "body": "### Incident Summary\n\n  \nOn May 6 at 11:50 PM PST, we deployed a configuration change to one of our core pipeline  \nservices. This change introduced an unintended interaction with our database layer, causing a  \nsignificant increase in write load. The resulting pressure degraded query and command  \nthroughput across the platform\n\n### Root Cause\n\n  \nThe configuration change introduced a blocking condition on expression evaluation in the  \npipeline service. When executions encountered blocked expressions, they failed and retried  \nrepeatedly, generating a write storm against the database and the  throughput went 4x\n\n‌\n\n### Remediation \n\n  \n● Rolled back the configuration change.  \n● Applied database-level tuning to reduce write pressure and accelerate backlog drainage  \n● Performed a controlled failover to a healthy database node to restore throughput  \n● Scaled up database nodes to provide sufficient capacity for full recovery\n\n‌\n\n### Preventive Actions\n\nTo prevent from such Issues happening again, we are focussing on:\n\n  \n1\\. Increased database resilience: We are implementing automated load-shedding thresholds  \nthat trigger on leading indicators \\(replication lag, session depth, op latency\\) before the database  \nreaches saturation, preventing retry storms from compounding into full degradation events.  \n\n2\\. We are optimizing databases so that we can increase write throughput by an order of magnitude and enable independent scaling of customer data workloads. This would have allowed us to drain the message backlog nearly instantaneously during this incident.",
        "createdAt": "2026-05-20T00:43:21.588+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}