Skip to content
OutageDeck

Incident detail

Project Actions Failing Across Multiple Regions

Project Actions Failing Across Multiple Regions

Active incidentMajor1 affected service

Timeline window

Started

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Investigating

    Certain project actions are failing. We are actively investigating to determine the regions impacted and source of the issue.

  2. Investigating

    Project actions in us-east-1, us-east-2, sa-east-1, ca-central-1 are experiencing significant increases in error rates. We are exploring the best options for restoring performance.

  3. Investigating

    We have determined that there are four main project actions impacted in the identified regions: Pause/unpause failures, Restore/PITR failures, PITR configuration failures and Backup problems. We are pursuing root cause and mitigation options in parallel.

  4. Investigating

    We are testing mitigation options to attempt to restore project action performance.

  5. Identified

    We have identified a problem with a specific job in us-east-1, we have mitigated the job and are observing the results.

  6. Monitoring

    The mitigation has returned project activity to normal performance. Project actions that failed during the outage are being requeued to run. We will continue to monitor to ensure project functionality.