{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Teams, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-10-02T23:59:29.299Z"
  },
  "data": {
    "id": "incident_statuspage_harness_w85lbwhgqf2n",
    "slug": "harness-prod2-cv-is-failing-with-errors-for-certain-customers-2026-09-29",
    "title": "Prod2: CV is failing with errors for all customers",
    "summary": "Prod2: CV is failing with errors for all customers",
    "status": "resolved",
    "stale": false,
    "severity": "minor",
    "startedAt": "2026-09-29T14:41:24.865+00:00",
    "updatedAt": "2026-10-02T21:51:05.743+00:00",
    "resolvedAt": "2026-09-29T18:49:30.339+00:00",
    "provider": {
      "slug": "harness",
      "name": "Harness"
    },
    "affectedServices": [
      {
        "slug": "harness-cicd",
        "name": "CI/CD"
      }
    ],
    "links": {
      "html": "/incidents/harness-prod2-cv-is-failing-with-errors-for-certain-customers-2026-09-29",
      "api": "/api/v1/incidents/harness-prod2-cv-is-failing-with-errors-for-certain-customers-2026-09-29",
      "providerHtml": "/providers/harness",
      "alerts": "https://outagedeck.com/account?stack=harness&utm_source=api&utm_medium=response&utm_campaign=api_alerts&utm_content=incident"
    },
    "impactSummary": "Harness reported a minor event for the affected tracked services.",
    "source": {
      "id": "source_harness_status",
      "kind": "official_api",
      "name": "Harness Status",
      "checkedAt": "2026-10-02T23:55:04.645+00:00",
      "stale": false,
      "officialUrl": "https://status.harness.io",
      "statusPageUrl": "https://status.harness.io"
    },
    "updates": [
      {
        "id": "update_statuspage_harness_w85lbwhgqf2n_546vy7sccnpj",
        "status": "investigating",
        "body": "We are currently investigating this issue.",
        "createdAt": "2026-09-29T14:41:25.127+00:00"
      },
      {
        "id": "update_statuspage_harness_w85lbwhgqf2n_djwqw5t4n4kn",
        "status": "monitoring",
        "body": "A fix has been implemented and we are monitoring the results.",
        "createdAt": "2026-09-29T15:54:15.301+00:00"
      },
      {
        "id": "update_statuspage_harness_w85lbwhgqf2n_3g3wf51hb047",
        "status": "monitoring",
        "body": "We are continuing to monitor for any further issues.",
        "createdAt": "2026-09-29T15:57:41.499+00:00"
      },
      {
        "id": "update_statuspage_harness_w85lbwhgqf2n_s280w37ns78s",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2026-09-29T18:49:30.339+00:00"
      },
      {
        "id": "update_statuspage_harness_w85lbwhgqf2n_nlvw8z6k3b6l",
        "status": "resolved",
        "body": "**Summary**\n\nCustomers on Prod-2 experienced elevated failure rates in Continuous Verification pipelines. Verification jobs failed to complete or reported errors because CV data-collection workers could not communicate with the platform service that coordinates their work. This was caused by an unexpected traffic spike to one of our backend systems. \n\nService was restored by scaling out the affected platform component. No data loss or corruption occurred.\n\n‌\n\n**Customer Impact**\n\nCustomers using Continuous Verification on Prod-2 saw verification pipeline jobs fail or time out during the incident window. A significant fraction of verification job instances that ran during this period ended in an operational failure. Pipelines not using CV continued to operate normally.\n\n‌\n\n**Root Cause**\n\nA large automated provisioning run created a significant number of persistent CV monitoring workers in a short window, which overwhelmed a shared platform component.\n\nThe sudden increase in concurrent outbound calls exhausted the available network connections on each service instance.  \n\n‌\n\n**Mitigation**\n\nThe maximum scaling limit for the CV service was raised, allowing it to expand capacity to meet demand. Failures stopped shortly after the scale-out was completed.\n\n‌\n\n**Next Steps**\n\nTo prevent recurrence, Harness will:\n\n* **Increase capacity headroom and add provisioning guardrails:** Raise the CV service scaling limit to maintain headroom under burst loads, and introduce rate limiting for bulk provisioning of monitored services.\n* **Improve Resiliency:** Add exponential backoff and jitter for worker retries, and use a reusable connection pool to reduce overhead and prevent transient failures from compounding under high concurrency.\n* **Improve monitoring and alerting:** Enhance alerting for abnormal per-account worker growth rates, so similar conditions are detected and acted on earlier.",
        "createdAt": "2026-10-02T05:37:52.786+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}