Provider
Fly.ioIncident detail
Upstash unavailability in FRA region
Resolved incidentMajor
Timeline window
to
Get alerted the next time Fly.io breaks
Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.
Timeline
Incident updates
Every update on the official status source, oldest to newest, exactly as it appeared there.
Investigating
We are investigating an issue affecting Upstash services in the fra region. Customers may experience connection failures or service unavailability. We are working with Upstash to identify the cause and restore service. We’ll provide an update as soon as we have more information.
Resolved
This incident has been resolved.
Monitoring
A fix has been implemented and we are monitoring the results.
Resolved
Impact
From about 12:00 to 13:05 UTC, some clients got connection timeouts to their Redis databases. Affected were databases hosted in fra or gig, and clients whose connections were routed through fra or gig, even if their database is hosted in another region.
What happened
During a planned rolling upgrade, replicas in fra and gig did not finish draining and were left out of service. They were returned to Fly routing before they were ready to accept connections. Connections routed to them then timed out.
What we're changing
• We will be adding tooling to return a replica to service safely after maintenance.
• We will be adding safety checks to our upgrade process.
• We will make the upgrade procedure more resilient to errors like this one.
Keep exploring
More from Fly.io
Neighboring incidents on Fly.io's timeline and the rest of their record on OutageDeck.