Skip to content

Incident detail

Some cypher query regression

Resolved incidentMinor2 affected services

Timeline window

to

Get alerted the next time Neo4j Aura breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Identified

    Aura 5 latest update has introduced a query regression

    Error Signature: key not found: VariableSlotKey(ke)

    Workaround: prefixing the queries with runtime=legacy should address the issue

  2. Identified

    We continue to work on a fix for this issue.

    To identify if you are affected - Error Signature: key not found: VariableSlotKey(...)

    We are taking steps to contact affected customers.

  3. Identified

    We have a fix and are currently ongoing packaging and release of that fix.

    The ETA for the rollout on Aura is currently 22:00 UTC

  4. Identified

    We are continuing to work on a fix for this issue.

  5. Identified

    We have a fix and are currently progressing with the release of that fix.

    The packaging and testing ran longer than earlier estimates.

    The current projected ETA for the rollout on Aura is currently June 29th 00:00 UTC

  6. Resolved

    We have now completed the roll out of the fix to all affected Aura instances. incident resolved

  7. Resolved

    ### What happened

    On 2024-06-28 10:46 (UTC) we released a new version of the database (v. 5.21). This release of the database contained an improvement to fix a problem of queries returning an entity ordered by an indexed-backed property that could come out in the wrong order due to concurrent writes (eg with an index-backed property:  `MATCH (n:L) WHERE n.x IS NOT NULL RETURN n ORDER BY n.x`) Unfortunately this introduced two undetected regressions:

    • `key not found: VariableSlotKey(...)` where the runtime would try to access a cached variable that was not in scope 
    • `NullCheckReferenceProperty cannot be cast to class ASTCachedProperty`  a class cast exception in the slotted runtime happening in some cases when retrieving cached properties

    Contrary to the usual process, a human error meant it was rolled-out directly to the AuraDB Enterprise tier without having been exposed first to the AuraDB Free tier and Professional tier.

    ### How the service was affected

    Customers running certain types of queries where the whole node would get cached as part of the optimization were seeing failures `key not found: VariableSlotKey(...)` OR  `NullCheckReferenceProperty cannot be cast to class ASTCachedProperty` with no simple workarounds.

    We fixed the regression and rolled out a new version 5.21.1 (2024-06-29 01:10) and then a second 5.21.2 to address both of these regressions.

    ### What we are doing now

    We consider this incident as severe and have committed resources to deliver on the following actions:

    • Rolling out
    • Prevent rolling out to the Enterprise tier at the end of the working week
    • Release roll-out process and tooling to enforce the sequence of tiers to receive a new release
    • Automate the release and roll-out process to remove risks of further human error
    • Improve on the time it takes to release an emergency fix
    • Database release
    • Add further tests around the release of improvements to Cypher
    • Detection
    • Build a monitoring dashboard specific to detect spikes of errors due to Cypher queries

Keep exploring

More from Neo4j Aura

Neighboring incidents on Neo4j Aura's timeline and the rest of their record on OutageDeck.