Skip to content

Incident detail

MTLS Outage with Delegates

Resolved incidentCritical1 affected service

Timeline window

to

Get alerted the next time Harness breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Investigating

    We are currently investigating this issue.

  2. Identified

    The issue has been identified and a fix is being implemented.

  3. Identified

    We are continuing to work on a fix for this issue.

  4. Monitoring

    A fix has been implemented and we are monitoring the results.

  5. Resolved

    This incident has been resolved.

  6. Resolved

    # Summary

    To improve the security posture, On September 14, 2026, Harness performed a scheduled rotation of the TLS certificate used to secure Delegate connections for accounts using mutual TLS (mTLS),The replacement

    certificate passed Harness's existing pre-deployment validation and smoke testing. However, the rotation also

    introduced a change in certificate authority, which required additional certificate-chain and gateway

    configuration this prevented affected Delegates from connecting to Harness.

    Harness has identified the contributing factors and is implementing the corrective actions described below to prevent a recurrence.

    # Impact

    Delegates configured to use mutual TLS in Prod environment were unable to connect to Harness for

    until Harness applied a temporary mitigation (disabling mTLS) that restored

    connectivity while the underlying issue was corrected.

    This affected the ability to run pipelines and other operations dependent on the affected Delegates. Full,

    permanent connectivity — with mTLS restored to its normal, fully secure configuration — was confirmed for

    both environments by 7:20 PM PST.

    # Root Cause

    ‌

    As part of routine certificate maintenance, Harness rotated the certificate used by the gateway component that

    secures mutual TLS connections for Delegates in prod environments. The rotation also introduced a

    change in certificate authority, which required additional certificate-chain and gateway configuration that was

    not fully covered by the existing validation process. As a result, the gateway was unable to load the new

    certificate correctly, and Delegates relying on mutual TLS could not establish a secure connection.

    # Remediation

    Once the certificate rotation was confirmed as the source of the issue, Harness engineering worked continuously

    to correct the certificate and validate the fix with Citi before fully restoring the secure configuration.

    Corrective and Preventive Actions

    During remediation, Harness also identified and corrected a secondary issue with the replacement certificate's

    chain, which required an additional deployment before connectivity was fully and permanently restored.

    # Next Steps

    Harness is taking the following steps to prevent a recurrence of this issue:

    1. Extend pre-deployment validation and smoke testing to explicitly cover certificate authority changes and full

    certificate-chain verification, closing the specific gap that allowed this configuration through.

    2. Enhance automation for managing mTLS-related configuration changes,

    3. Improve monitoring and alerting for Delegate connectivity, including account-level visibility, so similar issues

    are detected automatically

Keep exploring

More from Harness

Neighboring incidents on Harness's timeline and the rest of their record on OutageDeck.