Provider
Neo4j AuraIncident detail
AuraDB/DS : Some instance operations failing
Timeline window
to
Get alerted the next time Neo4j Aura breaks
Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.
Timeline
Incident updates
Every update on the official status source, oldest to newest, exactly as it appeared there.
Identified
We identified an issue with resuming or loading data on some instances. We are looking into the root cause.
Identified
We have identified the issue and are working on a fix. We will continue to update you as we work this problem.
Identified
We have identified the issue and are working on a fix. We will continue to update you as we work this problem.
Identified
We are currently working on a fix for the issue.
We will continue to update you as we work this problem.
Identified
We have identified the issue and are working on a fix.
We will continue to update you as we work on this problem.
Identified
We are currently working on a fix for the issue.
We will continue to update you as we work this problem.
Identified
We have identified the issue and are working on a fix.
We will continue to update you as we work on this problem.
Identified
We have identified the issue and are working on a fix.
We will continue to update you as we work on this problem.
Identified
We have identified the issue and are working on a fix.
We will continue to update you as we work on this problem.
Identified
We have identified the issue and are working on a fix.
We will continue to update you as we work on this problem.
Identified
We have identified the issue and are working on a fix.
We will continue to update you as we work on this problem.
Identified
We are currently working on a fix for the issue.
We will continue to update you as we work this problem.
Identified
We have identified the issue and are working on a fix.
We will continue to update you as we work on this problem.
Identified
We are currently working on a fix for the issue.
We will continue to update you as we work this problem.
Identified
We are currently working on a fix for the issue.
We will continue to update you as we work this problem.
Identified
Some operations (Resume, Restore from backup, Loading and Clone to new) on Aura instances may experience some failure rate.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Identified
Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.
We are working closely with our cloud provider on a solution.
Resolved
The issue is now fixed and our cloud provider has confirmed the resolution of the underlying root cause by rolling out a change.
Resolved
### What happened
Our cloud provider rolled out a new feature in GKE that handles the resources relating to kubernetes snapshots. A new configuration associated with this feature led to a race condition exacerbated by concurrent calls to the resource with identical snapshot names.
We worked to isolate the issue and to discard any involvement from our changes. After we got clarity, we declared this incident and engaged with our cloud service provider and reported the issue and escalated soon after as we became more confident of the nature and impact of the issue.
### How the service was affected
Most Aura operations (Resume, Restore from backup, Loading, and Clone to new) were failing as they require the kubernetes snapshot functionality. Some customers may have initiated operations and those could have got stuck. We proactively monitored and assisted manually whenever possible.
### What we are doing now
- Our provider is reviewing their testing procedure to detect better possible race conditions.
- Our provider is implementing additional safeguards to prevent race conditions from occurring.
- We have now established a better channel to work and escalate issues with our service provider when it is affecting our service.
Keep exploring
More from Neo4j Aura
Neighboring incidents on Neo4j Aura's timeline and the rest of their record on OutageDeck.