{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-07-24T10:19:00.348Z"
  },
  "data": {
    "id": "incident_statuspage_harness_6km5xhb74rqn",
    "slug": "harness-secret-resolution-failures-in-ci-executions-prod2-2026-05-12",
    "title": "Secret resolution failures in CI executions - PROD2",
    "summary": "Secret resolution failures in CI executions - PROD2",
    "status": "resolved",
    "severity": "major",
    "startedAt": "2026-05-12T09:50:29+00:00",
    "updatedAt": "2026-05-20T01:42:01.395+00:00",
    "resolvedAt": "2026-05-12T14:35:54+00:00",
    "provider": {
      "slug": "harness",
      "name": "Harness"
    },
    "affectedServices": [
      {
        "slug": "harness-cicd",
        "name": "CI/CD"
      }
    ],
    "links": {
      "html": "/incidents/harness-secret-resolution-failures-in-ci-executions-prod2-2026-05-12",
      "api": "/api/v1/incidents/harness-secret-resolution-failures-in-ci-executions-prod2-2026-05-12",
      "providerHtml": "/providers/harness"
    },
    "impactSummary": "Harness reported a major event for the affected tracked services.",
    "source": {
      "id": "source_harness_status",
      "kind": "official_status_page",
      "name": "Harness Status",
      "checkedAt": "2026-07-23T12:00:00Z",
      "officialUrl": "https://status.harness.io",
      "statusPageUrl": "https://status.harness.io"
    },
    "updates": [
      {
        "id": "update_statuspage_harness_6km5xhb74rqn_06kq4dnctvld",
        "status": "investigating",
        "body": "We are currently investigating this issue.",
        "createdAt": "2026-05-12T16:02:29.019+00:00"
      },
      {
        "id": "update_statuspage_harness_6km5xhb74rqn_wtc8xk1xl7cr",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2026-05-12T16:03:54.485+00:00"
      },
      {
        "id": "update_statuspage_harness_6km5xhb74rqn_34fx3c6m80zb",
        "status": "resolved",
        "body": "### Summary\n\nOn May 12, 2026, Harness identified an issue affecting Kubernetes-based CI builds in the prod2 environment. The issue was caused by the unintended enablement of an internal feature flag associated with a CI performance optimization.\n\nEffect was seen for build steps using older `ci-1.16.81` CI addon version \\(published 1 year ago\\). Affected builds using older CI addon versions experienced the following behavior during the incident window:\n\n* Secrets referenced in Run step commands resolved as empty values, causing build failures\n\nNo customer action is required.\n\n### Impact\n\n* Affected Environment: prod2\n* Affected Builds\n\n    * Kubernetes CI builds executed during the incident window\n    * Builds using CI addon versions older than `ci-1.16.81`\n    \n* Customer Impact: \n\n    * Secret expressions resolving as empty strings during execution\n    \n\n‌\n\n### Root Cause\n\nHarness introduced a CI optimization feature intended to improve Kubernetes pipeline execution performance. The feature depended on functionality available only in newer CI addon versions \\(`ci-1.16.81` and later\\).\n\nOn May 12, 2026, the feature flag controlling this optimization was unintentionally enabled for customer accounts in the prod2 environment because of misconfigured  checks. The intent of change was to verify in certain internal accounts as part of a phased roll out of this feature.\n\n‌\n\nFor environments running older addon versions:\n\n‌\n\n* Runtime secret resolution did not execute correctly, causing secrets referenced in Run step commands to resolve as empty values\n\n‌\n\nThe issue was mitigated immediately by disabling the feature flag globally.\n\n### Mitigation Actions\n\nHarness completed the following remediation steps:\n\n‌\n\n* Disabled the feature flag globally\n* Verified successful pipeline execution after rollback\n* Identified and deleted all affected downloadable log entries from GCP storage\n* Sent communications to affected customers\n\n### Preventive Actions\n\nHarness is implementing the following safeguards to prevent recurrence:\n\n* Enforcing addon version compatibility validation before enabling feature flags that depend on addon functionality\n* Documenting compatibility requirements between CI manager and addon versions",
        "createdAt": "2026-05-20T01:39:56.641+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}