Skip to content

Incident detail

AuraDB/DS : Some instance operations failing

Resolved incidentMinor1 affected service

Timeline window

to

Get alerted the next time Neo4j Aura breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Identified

    We identified an issue with resuming or loading data on some instances. We are looking into the root cause.

  2. Identified

    We have identified the issue and are working on a fix. We will continue to update you as we work this problem.

  3. Identified

    We have identified the issue and are working on a fix. We will continue to update you as we work this problem.

  4. Identified

    We are currently working on a fix for the issue.

    We will continue to update you as we work this problem.

  5. Identified

    We have identified the issue and are working on a fix.

    We will continue to update you as we work on this problem.

  6. Identified

    We are currently working on a fix for the issue.

    We will continue to update you as we work this problem.

  7. Identified

    We have identified the issue and are working on a fix.

    We will continue to update you as we work on this problem.

  8. Identified

    We have identified the issue and are working on a fix.

    We will continue to update you as we work on this problem.

  9. Identified

    We have identified the issue and are working on a fix.

    We will continue to update you as we work on this problem.

  10. Identified

    We have identified the issue and are working on a fix.

    We will continue to update you as we work on this problem.

  11. Identified

    We have identified the issue and are working on a fix.

    We will continue to update you as we work on this problem.

  12. Identified

    We are currently working on a fix for the issue.

    We will continue to update you as we work this problem.

  13. Identified

    We have identified the issue and are working on a fix.

    We will continue to update you as we work on this problem.

  14. Identified

    We are currently working on a fix for the issue.

    We will continue to update you as we work this problem.

  15. Identified

    We are currently working on a fix for the issue.

    We will continue to update you as we work this problem.

  16. Identified

    Some operations (Resume, Restore from backup, Loading and Clone to new) on Aura instances may experience some failure rate.

    We are working closely with our cloud provider on a solution.

  17. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  18. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  19. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  20. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  21. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  22. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  23. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  24. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.

    We are working closely with our cloud provider on a solution.

  25. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.

    We are working closely with our cloud provider on a solution.

  26. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.

    We are working closely with our cloud provider on a solution.

  27. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.

    We are working closely with our cloud provider on a solution.

  28. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.

    We are working closely with our cloud provider on a solution.

  29. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.

    We are working closely with our cloud provider on a solution.

  30. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not be completed successfully.

    We are working closely with our cloud provider on a solution.

  31. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  32. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  33. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  34. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  35. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution

  36. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  37. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  38. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  39. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  40. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  41. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  42. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  43. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  44. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  45. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  46. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  47. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  48. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  49. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  50. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  51. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  52. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  53. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  54. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  55. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  56. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  57. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  58. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  59. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  60. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  61. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  62. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  63. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  64. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  65. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  66. Identified

    Some operations (Resume, Restore from backup, Loading, and Clone to new) on Aura instances are impacted and may not complete successfully.

    We are working closely with our cloud provider on a solution.

  67. Resolved

    The issue is now fixed and our cloud provider has confirmed the resolution of the underlying root cause by rolling out a change.

  68. Resolved

    ### What happened

    Our cloud provider rolled out a new feature in GKE that handles the resources relating to kubernetes snapshots. A new configuration associated with this feature led to a race condition exacerbated by concurrent calls to the resource with identical snapshot names.

    We worked to isolate the issue and to discard any involvement from our changes. After we got clarity, we declared this incident and engaged with our cloud service provider and reported the issue and escalated soon after as we became more confident of the nature and impact of the issue.

    ### How the service was affected

    Most Aura operations (Resume, Restore from backup, Loading, and Clone to new) were failing as they require the kubernetes snapshot functionality. Some customers may have initiated operations and those could have got stuck. We proactively monitored and assisted manually whenever possible.

    ### What we are doing now

    • Our provider is reviewing their testing procedure to detect better possible race conditions. 
    • Our provider is implementing additional safeguards to prevent race conditions from occurring. 
    • We have now established a better channel to work and escalate issues with our service provider when it is affecting our service.

Keep exploring

More from Neo4j Aura

Neighboring incidents on Neo4j Aura's timeline and the rest of their record on OutageDeck.