Skip to content
OutageDeck

Incident detail

AWS Cluster DNS Changes Degraded Performance

AWS Cluster DNS Changes Degraded Performance

Resolved incidentMajor1 affected service

Timeline window

to

Sponsored placement

Google AdSense placeholder

This slot is reserved for future AdSense placement. It stays as a non-invasive placeholder until a client ID and slot ID are configured.

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Identified

    DNS changes for AWS clusters are currently degraded. We've identified the issue and are pushing out a patch.

  2. Identified

    DNS changes for AWS clusters are currently degraded. Cluster topology modifications may be delayed or not able to complete successfully. We've identified the issue and are pushing out a patch.

  3. Identified

    Mitigation is in progress; recovery for DNS operations and downstream cluster topology changes has begun.

  4. Investigating

    We are still seeing cases where new cluster update requests are failing to go through due to ACME related DNS issues. We are actively investigating this issue.

  5. Monitoring

    We've identified a bad piece of metadata causing DNS issues. We've since patched this metadata and are seeing recovery in ACME updates for cluster topology changes. We will continue to monitor recovery.

  6. Monitoring

    We are continuing to monitor for any further issues.

  7. Resolved

    DNS updates for ACME are confirmed healthy. Cluster updates are now proceeding as expected.