Skip to content
OutageDeck

Incident detail

Pipeline and secret resolution failures

Resolved incidentMinor

Timeline window

to

Outage alerts

Get alerted the next time Harness breaks

Free email alerts for up to 5 providers — no card, live in about a minute. Paid plans add Slack, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Set up free alertsHow Harness alerts work

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Resolved

    On May 8, 2026, a CI performance optimization feature flag was inadvertantly impacted customer with older delegate version. The feature was designed to improve CI pipeline initialization performance by deferring secret resolution to runtime on supported delegate versions

  2. Resolved

    On May 8, 2026, a CI performance optimization feature flag was inadvertently impacted customer with older delegate version. The feature was designed to improve CI pipeline initialization performance by deferring secret resolution to runtime on supported delegate versions.

    Customers running delegate versions that did not support the new runtime secret resolution capability experienced pipeline execution failures where secrets referenced in Run step commands resolved as empty strings.

    The issue primarily impacted customers running older delegate versions outside the currently supported delegate release window. Customers on supported delegate versions were not impacted.

    The issue was mitigated by disabling the feature flag, which immediately restored the prior secret resolution behavior. No customer configuration changes were required.

    Impact

    • Pipeline execution failures for affected Kubernetes CI builds
    • Secrets referenced in Run step commands resolved as empty strings
    • Customers running supported delegate versions were not affected

    Root Cause

    A feature flag intended for a limited internal rollout was unintentionally enabled for a broader set of accounts during a routine rollout process. The feature relied on delegate-side functionality available only in newer delegate versions. Older delegate versions did not support the required runtime secret resolution behavior, resulting in pipeline failures.

    ‌

    Resolution

    The feature flag was disabled globally, restoring the previous behavior where secrets are resolved during pipeline initialization. Impacted pipelines recovered immediately after rollback.

    Preventive Actions

    Harness is implementing the following improvements:

    • Enhancements to the feature flag approval workflow to more clearly display rollout scope.
    • Improved process and documentation for feature flags that depend on minimum delegate versions
    • Expanded monitoring and alerting for secret resolution failures in CI pipelines