Degraded networking to AWS IPs

We are investigating reports of elevated networking issues between our network and AWS IPs. We are working to determine the source of the issue.

Our initial investigations indicate connection errors are occuring the return path from AWS IPs. We are escalating the issue with AWS and continuing to investigate.

Customers seeing issues are encouraged to also contact AWS support or their AWS hosted vendors support with details of the connection errors and their fly machine IPs.

We are continuing to see connectivity issues from AWS IPs to a subset of our hosts. From our investigation these connections appear to be being blocked from the AWS side. We are continuing to work with our upstream networking partners and AWS on remediation.

In the meantime impacted customers are recommended to allocate app-scoped static egress IPs using fly ips allocate-egress.

We are continuing to work with our upstream networking partners and AWS on remediation. Impacted customers are recommended to allocate app-scoped static egress IPs using fly ips allocate-egress.

Customers seeing issues are encouraged to also contact AWS support or their AWS hosted vendors support with details of the connection errors and their fly machine IPs.

AWS has implemented changes to unblock connections from their network from reaching our machines. We are seeing connectivity starting to resume on a host-by-host basis as these changes take effect.

Connectivity from AWS to Fly.io on impacted hosts is continuing to recover. Approximately 80% of the hosts that were seeing connectivity issues are fully recovered at this time.

This incident has been resolved, connectivity from AWS to all of our hosts is back to normal.