Skip to content
OutageDeck

Incident detail

Hosted CI Builds Failing Intermittently

Resolved incidentMajor1 affected service

Timeline window

to

Outage alerts

Get alerted the next time Harness breaks

Free email alerts for up to 5 providers — no card, live in about a minute. Paid plans add Slack, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Set up free alertsHow Harness alerts work

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Investigating

    We are currently investigating this issue.

  2. Identified

    The issue has been identified and a fix is being implemented.

  3. Identified

    We are continuing to work on a fix for this issue.

  4. Identified

    Harness is continuing to investigate and implement changes to fully restore functionality. At this time, we are still seeing some CI Builds intermittently fail.

  5. Identified

    Harness has implemented a change and are seeing failures reduced, but are continuing to work on completing mitigation. Customers can expect executions to succeed more frequently.

    Prod1/Prod2 are back to normal.

  6. Identified

    Harness is currently implementing a failover to mitigate the intermittent issue for some customers

  7. Monitoring

    A fix has been implemented and we are monitoring the results.

  8. Resolved

    This incident has been resolved.

  9. Resolved

    ### Summary

    Between May 27 and June 1, 2026, some Harness CI customers experienced pipeline execution failures with the error:

    `failed to call LE.RetryStartStep: context deadline exceeded`

    ### Impact

    Affected customers saw intermittent CI pipeline failures during step execution.

    Existing pipeline definitions, customer data, source code, and artifacts were not impacted.

    ### Root Cause

    The root cause was a deadlock in the Light Engine logging path. When the log service returned an error, the Light Engine log writer attempted to reacquire a mutex it already held. This caused the Light Engine process to freeze, which led to step execution timeouts and pipeline failures.

    ### Mitigation and Resolution

    Harness Engineering took multiple mitigation steps during the incident, including:

    • Rolled back affected runner versions where needed
    • Increased relevant timeout configurations
    • Reduced log-service load and latency
    • Temporarily disabled the affected livelog streaming path
    • Migrated selected workloads across regions and infrastructure providers
    • Pinned a fixed Light Engine version through runner configuration

    The final fix addressed the mutex deadlock in the Light Engine log writer and prevented the same lock from being reacquired while already held.

    ### Prevention and Follow-Up Actions

    Harness is taking the following actions to reduce recurrence risk:

    • Improve deadlock detection in critical concurrent code paths
    • Strengthen error handling for log-service interactions
    • Add better monitoring for Light Engine process health
    • Improve safeguards around logging-path failures
    • Continue reviewing runner rollout and validation processes