Provider
Neo4j AuraIncident detail
Some cypher query regression
Timeline window
to
Get alerted the next time Neo4j Aura breaks
Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.
Timeline
Incident updates
Every update on the official status source, oldest to newest, exactly as it appeared there.
Identified
Aura 5 latest update has introduced a query regression
Error Signature: key not found: VariableSlotKey(ke)
Workaround: prefixing the queries with runtime=legacy should address the issue
Identified
We continue to work on a fix for this issue.
To identify if you are affected - Error Signature: key not found: VariableSlotKey(...)
We are taking steps to contact affected customers.
Identified
We have a fix and are currently ongoing packaging and release of that fix.
The ETA for the rollout on Aura is currently 22:00 UTC
Identified
We are continuing to work on a fix for this issue.
Identified
We have a fix and are currently progressing with the release of that fix.
The packaging and testing ran longer than earlier estimates.
The current projected ETA for the rollout on Aura is currently June 29th 00:00 UTC
Resolved
We have now completed the roll out of the fix to all affected Aura instances. incident resolved
Resolved
### What happened
On 2024-06-28 10:46 (UTC) we released a new version of the database (v. 5.21). This release of the database contained an improvement to fix a problem of queries returning an entity ordered by an indexed-backed property that could come out in the wrong order due to concurrent writes (eg with an index-backed property: `MATCH (n:L) WHERE n.x IS NOT NULL RETURN n ORDER BY n.x`) Unfortunately this introduced two undetected regressions:
- `key not found: VariableSlotKey(...)` where the runtime would try to access a cached variable that was not in scope
- `NullCheckReferenceProperty cannot be cast to class ASTCachedProperty` a class cast exception in the slotted runtime happening in some cases when retrieving cached properties
Contrary to the usual process, a human error meant it was rolled-out directly to the AuraDB Enterprise tier without having been exposed first to the AuraDB Free tier and Professional tier.
### How the service was affected
Customers running certain types of queries where the whole node would get cached as part of the optimization were seeing failures `key not found: VariableSlotKey(...)` OR `NullCheckReferenceProperty cannot be cast to class ASTCachedProperty` with no simple workarounds.
We fixed the regression and rolled out a new version 5.21.1 (2024-06-29 01:10) and then a second 5.21.2 to address both of these regressions.
### What we are doing now
We consider this incident as severe and have committed resources to deliver on the following actions:
- Rolling out
- Prevent rolling out to the Enterprise tier at the end of the working week
- Release roll-out process and tooling to enforce the sequence of tiers to receive a new release
- Automate the release and roll-out process to remove risks of further human error
- Improve on the time it takes to release an emergency fix
- Database release
- Add further tests around the release of improvements to Cypher
- Detection
- Build a monitoring dashboard specific to detect spikes of errors due to Cypher queries
Keep exploring
More from Neo4j Aura
Neighboring incidents on Neo4j Aura's timeline and the rest of their record on OutageDeck.