Skip to content
OutageDeck

Incident detail

Macaroon Auth + Machines API Issues

Macaroon Auth + Machines API Issues

Resolved incidentCritical2 affected services

Timeline window

to

Sponsored placement

Google AdSense placeholder

This slot is reserved for future AdSense placement. It stays as a non-invasive placeholder until a client ID and slot ID are configured.

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Investigating

    We are investigating issues with Macaroon based authentication. This is impacting parts of the Machines API, Fly.io Dashboard, some flyctl operations and other platform features that rely on this.

  2. Identified

    We are continuing to address this issue. Platform authentication with macaroon based tokens is currently failing. Platform features that authenticate with macaroons including Machines API operations, Dashboard logins, some flyctl commands, fly-metrics.net Grafana, and deployments are failing at this time. Existing, running customer applications and machines remain reachable and running. We will provide another update within 15 minutes

  3. Identified

    We have identified the cause of the issue and are working on deploying a fix. Impacted features remain unavailable or degraded at this time. Already running customer applications/machines remain available. MPG clusters remain generally reachable and healthy, however new clusters cannot be provisioned and failovers may not complete. We will provide another update within 15 minutes.

  4. Identified

    An initial fix has been deployed and we are starting to see platform features recover. Users may still see degraded performance and intermittent failures at this time. We are continuing to address the issue to ensure a full stable recovery.

  5. Identified

    We continue to seeing degraded performance and increased errors with the Machines API and other platform features at this time. We are continuing to work on fully restoring service.

  6. Identified

    We are seeing elevated cluster errors with Managed Postgres clusters as the MPG control plane recovers from the API outage. MPG Users may see elevated rates of failing or slow connections, as well as increased primary/replica failovers. The managed postgres team is addressing any degraded clusters. We will provide a further update within 15m.

  7. Identified

    We have deployed another change and are seeing wider improvements in platform stability across all regions. Performance is trending to normal, though users may still see some degradation at this time. We are continuing to closely monitor to ensure full, stable recovery. We will provide another update in 15m.

  8. Monitoring

    A fix has been implemented and we are monitoring the results.

  9. Resolved

    This incident has been resolved and we are seeing all platform functions operate normally.