Elevated API Errors

We are investigating an issue with our API.

We apologize for the inconvenience and will share an update once we have more information.

We're continuing to investigate this issue to identify the root cause. We'll share another update soon.

We've identified the root cause, and we're seeing services are beginning to recover. We'll continue to monitor closely and share updates.

Our initial remediations are in place, and services have recovered. We're continuing to monitor closely.

Our remediations continue to be effective, and services continue to be recovered. We will continue further investigation and additional preventative measures.

On September 18, between 22:40–23:03 UTC, WorkOS experienced a partial outage affecting authentication and other API requests.

A failover of the cache used for feature flag evaluation left app connections stalled, and requests waiting on flag checks held database connections open, causing queues to build and other requests to fail.

Many of you experienced failures, including failed sign-ins and long delays. We're sorry for the disruption this caused. We're taking this event seriously and have already fixed the root cause, with work underway on additional layers of protection.

We will publish a full RCA next week explaining in detail what failed, why recovery was not automatic (should have been ~seconds), and what we're changing to prevent this issue from happening again.