Skip to content

Incident detail

Login errors in US region

Resolved incidentMajor1 affected service

Timeline window

to

Get alerted the next time Keeper Security breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Identified

    We are aware of a high number of errors in the US region. The issue will be fixed shortly.

  2. Resolved

    We have resolved the issue and plan to deploy a backend update shortly.

  3. Resolved

    At 9:21AM PST we received notifications of API errors coming from our backend systems. The issue resolved itself without any required action at 9:31AM PST. During that 10 minute period, there was a database lock in the primary writer instance causing new login sessions to fail.

    Recently, Keeper upgraded all production RDS Aurora databases to MySQL 8.x. This updated database engine is sensitive to one particular SQL query that was causing large table scans to occur. The code changes and QA process was already under way when this issue occurred. The backend code change which addresses this issue passed QA at 12:48PM PST and we released the fix to prevent this issue from occurring again (Release Jira ticket REL-4633). We’re keeping a close eye and will issue additional updates as necessary.

Keep exploring

More from Keeper Security

Neighboring incidents on Keeper Security's timeline and the rest of their record on OutageDeck.