Skip to content

Incident detail

Vault login errors

Resolved incidentMinor

Timeline window

to

Get alerted the next time Keeper Security breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Resolved

    Between 5:15PM PST and 5:30PM PST, there were a high number of Vault Login errors.

  2. Resolved

    ### Incident Summary

    The Keeper DevOps team was alerted to API errors in the US region. Upon investigation, we identified a high number of database deadlocks affecting performance. To mitigate the immediate impact, we restarted the affected writer instance.

    ### Root Cause

    Historically, similar issues were linked to specific queries impacting MySQL 8.x updates on the RDS Aurora platform. In this case, we discovered that a utility designed to monitor deadlocks and capture diagnostic data was inadvertently contributing to the problem. A known bug in the MySQL minor version used in production caused our data-gathering query—intended to analyze deadlocks—to trigger additional locking, exacerbating the issue.

    ### Resolution & Mitigation

    • We promptly disabled the problematic monitoring utility and applied database parameter optimizations in collaboration with the AWS team to enhance performance under peak load.
    • These changes were fully implemented within 24 hours, and no further excessive row locks have been observed.
    • Our DBA team is actively working with backend engineers to further optimize write operations for improved performance.
    • A backend code update with additional optimizations is scheduled for release the week of March 17.

    We appreciate your patience and will continue to monitor and enhance system performance to prevent similar issues in the future.

Keep exploring

More from Keeper Security

Neighboring incidents on Keeper Security's timeline and the rest of their record on OutageDeck.