{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Teams, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-10-21T06:09:38.619Z"
  },
  "data": {
    "id": "incident_statuspage_harness_t7lqrzsp8drk",
    "slug": "harness-prod2-was-intermittently-unavailable-2026-09-05",
    "title": "Prod2 was intermittently unavailable",
    "summary": "Prod2 was intermittently unavailable",
    "status": "resolved",
    "stale": false,
    "severity": "minor",
    "startedAt": "2026-09-05T09:40:02.532+00:00",
    "updatedAt": "2026-09-09T18:22:04.182+00:00",
    "resolvedAt": "2026-09-05T09:40:48.672+00:00",
    "provider": {
      "slug": "harness",
      "name": "Harness"
    },
    "affectedServices": [
      {
        "slug": "harness-platform",
        "name": "Platform & dashboards"
      }
    ],
    "links": {
      "html": "/incidents/harness-prod2-was-intermittently-unavailable-2026-09-05",
      "api": "/api/v1/incidents/harness-prod2-was-intermittently-unavailable-2026-09-05",
      "providerHtml": "/providers/harness",
      "alerts": "https://outagedeck.com/account?stack=harness&utm_source=api&utm_medium=response&utm_campaign=api_alerts&utm_content=incident"
    },
    "impactSummary": "Harness reported this event with no impact level.",
    "source": {
      "id": "source_harness_status",
      "kind": "official_api",
      "name": "Harness Status",
      "checkedAt": "2026-10-21T06:00:13.356+00:00",
      "stale": false,
      "officialUrl": "https://status.harness.io",
      "statusPageUrl": "https://status.harness.io"
    },
    "updates": [
      {
        "id": "update_statuspage_harness_t7lqrzsp8drk_3mr38rwjfc7w",
        "status": "investigating",
        "body": "We are currently investigating this issue.",
        "createdAt": "2026-09-05T09:40:02.595+00:00"
      },
      {
        "id": "update_statuspage_harness_t7lqrzsp8drk_9ksxml6tz3lc",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2026-09-05T09:40:48.672+00:00"
      },
      {
        "id": "update_statuspage_harness_t7lqrzsp8drk_lfxkrg02h8ln",
        "status": "resolved",
        "body": "## **Summary**\n\nBetween 12:34am PST and 12:38am PST on 5th September, the Delegate service manager experienced some elevated exceptions when attempting to write to the database. Consequently, delegate connections were dropped, causing them to disconnect. Delegate automatically re-attempts registration back to the `delegate service manager`  and majority of the delegates got connected back after the incident. For Docker and ECS delegates the automatic restart is not enabled unless these delegates have health monitoring enabled. For these delegates a manual restart is needed and was recommended.  Post restart the delegate would re-connect and the issue was resolved.\n\n## **Root cause**\n\nOn Prod2 cluster we identified a performance bottleneck in the delegate service that, under certain conditions, can increase database write latency and delay heartbeat processing which leads to delegates being disconnected.\n\n## **Impact**\n\nAll K8s delegates and \\`Docker/ECS\\` delegates got connected back immediately within 4 mins and started to function normally. The impact can be scoped to those specific types of delegates that didn’t have health monitoring enabled. \n\n## **Remediation**\n\n* Immediate: We have added additional monitoring and increased resources for handling the influx of traffic. \n* Permanent: We have identified a hotspot in the code that can cause high latency when writing to a database which we are actively working on resolving. \n\n## **Action Items**\n\nTo prevent such issues from happening again, Harness will work on the following:\n\n1. Increased targeted monitoring and alerting to initiate timely mitigation and prevent this from happening again.\n2. Fix the identified delegate service managers database client reconnect failures\n3. Fix the hotpots that can cause query latency.",
        "createdAt": "2026-09-09T18:22:01.901+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}