{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-07-24T10:18:27.915Z"
  },
  "data": {
    "id": "incident_statuspage_harness_743r4kg80xj7",
    "slug": "harness-hosted-ci-customers-using-secure-connect-are-facing-connectivity-issues-2026-05-12",
    "title": "Hosted CI customers using secure connect are facing connectivity issues",
    "summary": "Hosted CI customers using secure connect are facing connectivity issues",
    "status": "resolved",
    "severity": "minor",
    "startedAt": "2026-05-12T21:38:03.658+00:00",
    "updatedAt": "2026-05-20T02:04:00.533+00:00",
    "resolvedAt": "2026-05-12T23:10:39.274+00:00",
    "provider": {
      "slug": "harness",
      "name": "Harness"
    },
    "affectedServices": [
      {
        "slug": "harness-cicd",
        "name": "CI/CD"
      }
    ],
    "links": {
      "html": "/incidents/harness-hosted-ci-customers-using-secure-connect-are-facing-connectivity-issues-2026-05-12",
      "api": "/api/v1/incidents/harness-hosted-ci-customers-using-secure-connect-are-facing-connectivity-issues-2026-05-12",
      "providerHtml": "/providers/harness"
    },
    "impactSummary": "Harness reported a minor event for the affected tracked services.",
    "source": {
      "id": "source_harness_status",
      "kind": "official_status_page",
      "name": "Harness Status",
      "checkedAt": "2026-07-23T12:00:00Z",
      "officialUrl": "https://status.harness.io",
      "statusPageUrl": "https://status.harness.io"
    },
    "updates": [
      {
        "id": "update_statuspage_harness_743r4kg80xj7_dlkkrgn264w4",
        "status": "investigating",
        "body": "We are currently investigating this issue.",
        "createdAt": "2026-05-12T21:38:03.849+00:00"
      },
      {
        "id": "update_statuspage_harness_743r4kg80xj7_4wm1tzj56y0c",
        "status": "identified",
        "body": "The issue has been identified and a fix is being implemented.",
        "createdAt": "2026-05-12T21:40:43.505+00:00"
      },
      {
        "id": "update_statuspage_harness_743r4kg80xj7_357x64cwjfbx",
        "status": "monitoring",
        "body": "A fix has been implemented and we are monitoring the results.",
        "createdAt": "2026-05-12T22:50:40.807+00:00"
      },
      {
        "id": "update_statuspage_harness_743r4kg80xj7_2pqsrgvj17jr",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2026-05-12T23:10:39.274+00:00"
      },
      {
        "id": "update_statuspage_harness_743r4kg80xj7_1gpxn1y5khf9",
        "status": "resolved",
        "body": "## Summary\n\nOn May 12, 2026, Harness's Secure Connect service experienced a significant disruption that affected customers using Secure Connect for git connector checks and CI pipeline execution. Affected customers experienced connection timeouts and failures when attempting to use Secure Connect-enabled connectors, while connectors operating without Secure Connect continued to function normally.\n\nThe disruption was caused by an automated infrastructure maintenance event in our Google Kubernetes Engine \\(GKE\\) environment that left a critical load balancer in an unreachable state. The issue was resolved by recreating the affected load balancer and updating the corresponding DNS record to restore connectivity.\n\n**Impact:**\n\n* Secure Connect-enabled git connector tests failing with connection timeouts\n* CI pipelines hanging or failing when Secure Connect was enabled\n* Connectors operating without Secure Connect were unaffected\n\n## Root Cause\n\nThe disruption was triggered by an automated GKE \\(Google Kubernetes Engine\\) control plane upgrade on the cluster hosting our internal Secure Connect FRPS \\(Fast Reverse Proxy Server\\) infrastructure.\n\nThe upgrade executed in two sequential phases in rapid succession. Each phase restarts the GKE cloud controller manager, which is responsible for managing the lifecycle of Kubernetes LoadBalancer service IP addresses. As part of its normal reconciliation process, the controller performs a delete-then-re-insert cycle on IP address reservations when it restarts.\n\nThe second phase of the upgrade began before the first reconciliation cycle had fully completed. This created a race condition where the controller deleted the IP address reservation for the internal FRPS load balancer but was interrupted before it could re-acquire it.  No Harness application deployments caused or contributed to the issue.\n\n‌\n\n## Mitigation\n\nThe following steps were taken to restore service:\n\n1. **FRPS pod restart** — An initial rollout restart of the FRPS deployment was performed. This restored client connectivity to the FRPS server but did not resolve the underlying IP orphaning issue.\n2. **Load balancer recreation** — The internal Kubernetes service \\(`frps-internal`\\) was deleted and recreated, triggering GKE to provision a new load balancer with a new IP address \n3. **DNS record update** — The Cloud DNS A record for  was updated to point to the new load balancer IP. This restored end-to-end routing for all Secure Connect traffic.\n\nFull connectivity was confirmed via internal testing and subsequently verified by affected customers.\n\n## Preventative Actions\n\nHarness is committed to preventing this class of incident from recurring. The following actions are being implemented:",
        "createdAt": "2026-05-20T01:56:20.723+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}