Skip to content
OutageDeck

Incident detail

Secret resolution failures in CI executions - PROD2

Resolved incidentMajor1 affected service

Timeline window

to

Outage alerts

Get alerted the next time Harness breaks

Free email alerts for up to 5 providers — no card, live in about a minute. Paid plans add Slack, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Set up free alertsHow Harness alerts work

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Investigating

    We are currently investigating this issue.

  2. Resolved

    This incident has been resolved.

  3. Resolved

    ### Summary

    On May 12, 2026, Harness identified an issue affecting Kubernetes-based CI builds in the prod2 environment. The issue was caused by the unintended enablement of an internal feature flag associated with a CI performance optimization.

    Effect was seen for build steps using older `ci-1.16.81` CI addon version (published 1 year ago). Affected builds using older CI addon versions experienced the following behavior during the incident window:

    • Secrets referenced in Run step commands resolved as empty values, causing build failures

    No customer action is required.

    ### Impact

    • Affected Environment: prod2
    • Affected Builds
    • Kubernetes CI builds executed during the incident window
    • Builds using CI addon versions older than `ci-1.16.81`
    • Customer Impact:
    • Secret expressions resolving as empty strings during execution

    ‌

    ### Root Cause

    Harness introduced a CI optimization feature intended to improve Kubernetes pipeline execution performance. The feature depended on functionality available only in newer CI addon versions (`ci-1.16.81` and later).

    On May 12, 2026, the feature flag controlling this optimization was unintentionally enabled for customer accounts in the prod2 environment because of misconfigured checks. The intent of change was to verify in certain internal accounts as part of a phased roll out of this feature.

    ‌

    For environments running older addon versions:

    ‌

    • Runtime secret resolution did not execute correctly, causing secrets referenced in Run step commands to resolve as empty values

    ‌

    The issue was mitigated immediately by disabling the feature flag globally.

    ### Mitigation Actions

    Harness completed the following remediation steps:

    ‌

    • Disabled the feature flag globally
    • Verified successful pipeline execution after rollback
    • Identified and deleted all affected downloadable log entries from GCP storage
    • Sent communications to affected customers

    ### Preventive Actions

    Harness is implementing the following safeguards to prevent recurrence:

    • Enforcing addon version compatibility validation before enabling feature flags that depend on addon functionality
    • Documenting compatibility requirements between CI manager and addon versions