Skip to content

Incident detail

Several operations degraded for AWS instances in region us-east-1

Resolved incidentMinor2 affected services

Timeline window

to

Get alerted the next time Neo4j Aura breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Identified

    Many core operations for instances in AWS region us-east-1 are degraded as a result of AWS incident in that region: https://health.aws.amazon.com/health/status

    Core operations include create, pause, resume, clone, backup, among others.

    We continue to monitor the AWS incident and will provide updates accordingly.

  2. Identified

    We are continuing to monitor the AWS incident, and can report some slow improvement of impacted Aura instances in the AWS us-east1 region, though the service remains degraded in this region at this time.

    We are continuing to monitor progress in the AWS incident: https://health.aws.amazon.com/health/status

  3. Resolved

    Instance operations in AWS us-east-1 region are back to normal following the resolution of the AWS incident. We will continue to monitor but all data indicates that the issue is fully resolved at this time.

  4. Resolved

    ### What Happened

    Between 08:41 UTC on October 20th and 09:00 UTC on October 21st, Neo4j Aura experienced service disruptions affecting the us-east-1 (N. Virginia) region in AWS. The incident was triggered by a broad AWS regional outage impacting Identity and Access Management (IAM) and the EC2 control plane. This resulted in delayed backups, temporary loss of database fault tolerance for a subset of users, and internal delays in administrative actions due to toolchain failures.

    A detailed summary of the AWS regional incident can be found here: https://aws.amazon.com/message/101925/

    ### How the service was affected

    The primary cause was several AWS regional service disruptions in us-east-1. We will cover how each of these affected Neo4j Aura and its users. Neo4j Aura is designed to isolate regional failures. This is achieved through deploying customer instances in Orchestras that have instances in three availability zones and no cross region dependencies.

    AWS IAM and Identity Center became unresponsive, preventing Neo4j Aura’s automated systems from authenticating with AWS resources in us-east-1. This affected Neo4j Aura backups to be written to AWS S3 buckets. Customers were also not able to resume paused instances during this period as the resume process was not able to authenticate with AWS S3 buckets to retrieve the paused data set. Neo4j Aura’s inability to authenticate with Route53 to create new DNS entries affected Neo4j Aura DB creation, as new databases were created.

    AWS Network Load Balancer health check system failures and AWS EC2 “request limit exceeded” or “insufficient capacity” errors in us-east-1 were false negatives with the NLB heath checks which resulted in some Neo4j Aura DB clusters in us-east-1 losing fault tolerance (1 out of 3 cluster members unavailable). When this happened instances were removed by kubernetes and we were not able to provision new ones due to the EC2 failures noted earlier, leaving the clusters without fault tolerance for a prolonged period of time. All of these clusters still had full availability of the other two cluster members.

    ### What are we doing now

    Largely Neo4j Aura responded as designed to these events, isolating failures to us-east-1, with no to minimal cross system failure propagation. The cross cloud impact of creating DNS records for new instances was the main outlier.

    To mitigate the scope of impact of future regional outages, Neo4j is implementing the following measures:

    • Full Neo4j Aura reviews to ensure Neo4j Aura design is implemented as intended across all sub-systems to surface any cross region dependencies. 
    • Remove identified cross-region dependencies.

Keep exploring

More from Neo4j Aura

Neighboring incidents on Neo4j Aura's timeline and the rest of their record on OutageDeck.