Skip to content

Incident detail

We are noticing CI hosted build failures

Resolved incidentMinor1 affected service

Timeline window

to

Outage alerts

Get alerted the next time Harness breaks

Free email alerts for up to 5 providers — no card, live in about a minute. Paid plans add Slack, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Investigating

    We are currently investigating this issue.

  2. Identified

    The issue has been identified and a fix is being implemented.

  3. Identified

    We are continuing to work on a fix for this issue.

  4. Monitoring

    A fix has been implemented and we are monitoring the results.

  5. Resolved

    This incident has been resolved.

  6. Resolved

    On April 16, 2026, Hosted CI Linux pipelines experienced intermittent initialization failures due to an upstream outage affecting package repositories used during environment setup.

    ### Impact

    A subset of customers running Hosted CI pipelines encountered failures during the initialization phase, preventing jobs from starting successfully.

    • Affected: 24 accounts
    • Failed executions: 129 pipelines
    • Impact duration: ~5 hours 49 minutes

    ### Root Cause

    The issue was caused by a service disruption in an external package repository provider. During CI environment provisioning, dependency installation requests to this upstream service timed out, causing initialization failures.

    ### Remediation

    Immediate Mitigation

    We updated runner configurations to bypass dependency installation during initialization and rolled out updated environments across affected clusters.

    Permanent Fix

    We have improved resilience in our CI infrastructure by:

    • Using pre-configured environments with required dependencies pre-installed
    • Eliminating runtime dependency on external package repositories during initialization
    • Enhancing failure handling for external dependency timeouts

    ### Action Items / Next Steps

    • Continue improving isolation from external dependencies during environment startup
    • Strengthen monitoring and alerting for upstream service degradation
    • Optimize rollout speed for infrastructure changes to reduce mitigation time