Skip to content

Incident detail

Admin functions impacted when accessing Users and AD Sync on certain deployments

Resolved incidentCritical2 affected services

Timeline window

to

Get alerted the next time Cisco Duo breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Investigating

    We are currently investigating an issue impacting user management and Active Directory sync on selected deployments. Please refer to the status page for further updates.

  2. Investigating

    We are continuing to investigate this issue.

  3. Identified

    A fix has been deployed for the issue affecting User Management and SSO logins on Duo3, Duo47, and Duo57. We are monitoring performance and stability. Please refer to the status page for further updates.

  4. Monitoring

    Following the implementation of the fix, we are seeing signs of recovery. We will continue to actively monitor the results to ensure the issue is fully resolved.

    Please refer to the status page for further updates.

  5. Resolved

    The issue affecting User Management and SSO logins on Duo3, Duo47, and Duo57 has been fully resolved, and all services have been restored to normal functionality.

    We will publish a RCA as soon as it is available. You can check back here for the latest information or subscribe for updates.

  6. Resolved

    Summary

    A change in the D322 build triggered high database load by calling an unoptimized query. This resulted in disruptions across several deployments. The system was stabilized after recovery actions.

    Timeline of Events

    08/26/2025 06:47:00 UTC - Alert fires that our service latency is above target.

    08/26/2025 06:58:00 UTC - On-call engineer confirms that autoscaling is in progress.

    08/26/2025 07:00:00 UTC - Separate alert fires that our service database CPU utilization is high, and requests backlog starts to grow.

    08/26/2025 07:37:00 UTC - On-call engineer disables certain features of our service to alleviate load.

    08/26/2025 08:08:00 UTC - Status page is published.

    08/26/2025 08:45:00 UTC - Service database is scaled up.

    08/26/2025 08:50:00 UTC - On-call engineer monitors recovery of systems.

    08/26/2025 10:40:00 UTC - All features of our service are reenabled.

    08/26/2025 11:44:00 UTC - Status page is updated to resolved.

    Details

    Our D322 build started calling an unoptimized API endpoint on an internal system which led to high database load, latency and request timeouts. This degraded multiple deployments (DUO3, DUO47, DUO57).

    ‌

    To alleviate load, certain features were disabled temporarily. However, because Duo Directory depends on this feature, customers received errors until it was re-enabled. At the same time, the database was scaled up vertically to allow for quicker processing of backlogged requests.

    ‌

    Once the backlog was cleared, all features were reenabled and the incident was resolved.

    ‌

    Deep analysis was conducted to determine why the service database CPU utilization was higher than expected. A newly introduced unoptimized query was found and fixed promptly.

    ‌

    Furthermore, our teams have plans to implement more rigorous testing and review processes for new database queries that are added to our services.

    ‌

    Also, our teams will work to improve our performance testing capabilities in the development environment to allow more accurate simulation of production load. This will aid us in being able to better detect unoptimized queries before they are released.

Keep exploring

More from Cisco Duo

Neighboring incidents on Cisco Duo's timeline and the rest of their record on OutageDeck.