Skip to content

Incident detail

2026-September-28 Service Incident

Resolved incidentMajor1 affected service

Timeline window

to

Get alerted the next time Sauce Labs breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Investigating

    We are currently seeing elevated error rates for automated real device tests in US-West-1 data center. We are currently investigating.

  2. Monitoring

    The root cause of the elevated error rates for Real Device tests in the US-West-1 data center has been identified. Remediation has been applied, and we are actively monitoring the situation.

  3. Resolved

    After taking remedial action, Real Device test error rates have returned to normal in the US-West-1 data center. All services are fully operational.

  4. Resolved

    ### Dates:

    Monday, September 28th 2026, 14:19 UTC - 16:52 UTC.

    ### What happened:

    Real device tests in US West hung and failed to output test artifacts due to slow uploads.

    ### Why it happened:

    A degraded interconnect link suffered high packet loss from optical errors, but BGP stayed up, preventing automatic failover to healthy paths.

    ### How we fixed it:

    Manually disabled the degraded port on the edge router, forcing traffic onto healthy alternate interconnects.

    ### What we are doing to prevent it from happening again:

    Automating link draining based on CRC error thresholds, improving protocols for isolating flapping vendor links, and hardening upload resilience.

Keep exploring

More from Sauce Labs

Neighboring incidents on Sauce Labs's timeline and the rest of their record on OutageDeck.