Skip to content

Incident detail

Slowness in Prod1 and Prod2 environment

Resolved incidentMinor3 affected services

Timeline window

to

Get alerted the next time Harness breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Investigating

    We are currently investigating this issue.

  2. Investigating

    We are continuing to investigate this issue.

  3. Investigating

    We are continuing to investigate the issue.

  4. Investigating

    We are continuing to investigate this issue.

  5. Identified

    The issue has been identified and a fix is being implemented.

  6. Identified

    We are continuing to work on a fix for this issue.

  7. Monitoring

    A fix has been implemented and we are monitoring the results.

  8. Monitoring

    A fix has been implemented and we are monitoring the results.

  9. Resolved

    This incident has been resolved.

  10. Resolved

    ### Summary

    On September 8, 2026, customers in Prod 1 and Prod 2 experienced elevated platform latency and pipeline failures. The issue was caused by a regression in a newly released capability that triggered cascading failures under high load. Because the capability was behind a feature flag, it was quickly disabled, and service was restored after a brief monitoring period.

    ### Customer Impact

    • Customers encountered slowness and failures during pipeline execution and UI operations. Some API calls returned errors or timed out.
    • No data loss or corruption occurred.

    ### Root Cause

    The new capability introduced a regression that created contention on a shared backend resource used by multiple Harness components. This saturated the shared platform infrastructure and caused the cascading failures.

    ### Mitigation

    • Disabled the capability across all environments
    • Temporarily increased platform capacity to restore stability

    ### Next Steps

    To prevent recurrence, Harness will:

    1. Permanently fix the capability by profiling and eliminating the sub-optimal code path and query

    2. Improve detection by enhancing alerting for resource-intensive queries on high-frequency platform paths

Keep exploring

More from Harness

Neighboring incidents on Harness's timeline and the rest of their record on OutageDeck.