Skip to content

Incident detail

Aura operations on AWS affected by an incident in US-EAST-1 region

Resolved incidentMinor2 affected services

Timeline window

to

Get alerted the next time Neo4j Aura breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Identified

    An active issue on AWS us-east-1 is affecting Neo4j Aura operations in the regions where we have some dependency on an impacted AWS service.

    See more details https://health.aws.amazon.com/health/status

    Other regions not currently impacted.

  2. Identified

    Operations such as Create, Pause, Resume, Clone are affected globally due to a dependency. Backups in the affected region of AWS us-est-1 will be impacted

    The affected AWS service is now starting to recover and we are monitoring the situation

  3. Monitoring

    We see clear signs of recovery and our impacted service are progressively getting back to normal.

    We will monitor and update when we can confirm full recovery and normal operations.

  4. Monitoring

    We are continuing to see good recovery across the service. We will update when this is completed.

  5. Resolved

    The underlying AWS incident has bene resolved and our services have now resumed and fully recovered

  6. Resolved

    ### What Happened

    Between 08:41 UTC on October 20th and 09:00 UTC on October 21st, Neo4j Aura experienced service disruptions affecting the us-east-1 (N. Virginia) region in AWS. The incident was triggered by a broad AWS regional outage impacting Identity and Access Management (IAM) and the EC2 control plane. This resulted in delayed backups, temporary loss of database fault tolerance for a subset of users, and internal delays in administrative actions due to toolchain failures.

    A detailed summary of the AWS regional incident can be found here: https://aws.amazon.com/message/101925/

    ### How the service was affected

    The primary cause was several AWS regional service disruptions in us-east-1. We will cover how each of these affected Neo4j Aura and its users. Neo4j Aura is designed to isolate regional failures. This is achieved through deploying customer instances in Orchestras that have instances in three availability zones and no cross region dependencies.

    AWS IAM and Identity Center became unresponsive, preventing Neo4j Aura’s automated systems from authenticating with AWS resources in us-east-1. This affected Neo4j Aura backups to be written to AWS S3 buckets. Customers were also not able to resume paused instances during this period as the resume process was not able to authenticate with AWS S3 buckets to retrieve the paused data set. Neo4j Aura’s inability to authenticate with Route53 to create new DNS entries affected Neo4j Aura DB creation, as new databases were created.

    AWS Network Load Balancer health check system failures and AWS EC2 “request limit exceeded” or “insufficient capacity” errors in us-east-1 were false negatives with the NLB heath checks which resulted in some Neo4j Aura DB clusters in us-east-1 losing fault tolerance (1 out of 3 cluster members unavailable). When this happened instances were removed by kubernetes and we were not able to provision new ones due to the EC2 failures noted earlier, leaving the clusters without fault tolerance for a prolonged period of time. All of these clusters still had full availability of the other two cluster members.

    ### What are we doing now

    Largely Neo4j Aura responded as designed to these events, isolating failures to us-east-1, with no to minimal cross system failure propagation. The cross cloud impact of creating DNS records for new instances was the main outlier.

    To mitigate the scope of impact of future regional outages, Neo4j is implementing the following measures:

    • Full Neo4j Aura reviews to ensure Neo4j Aura design is implemented as intended across all sub-systems to surface any cross region dependencies. 
    • Remove identified cross-region dependencies.

Keep exploring

More from Neo4j Aura

Neighboring incidents on Neo4j Aura's timeline and the rest of their record on OutageDeck.