[Serverless Functions/Containers] - [nl-ams] - Connectivity issues

The scheduled maintenance https://status.scaleway.com/incidents/f7lpf0j1c2sq did not go as planned. Since 09:50 UTC, Serverless Containers services might have faced connectivity issues. Create and update operations were impacted as well.

Serverless Container, Serverless Functions are impacted.

Connectivity has been back to normal since 11:06 UTC. We are continuing to monitor the situation.

We are still working on restoring create and update operations, which remain impacted.

As of 12:00 UTC, containers shouldn't be stuck anymore in "creating" or "updating" status for an abnormal duration.

Some containers with VPC support might have lost connectivity to resources inside the Private Network. A redeployment of the container triggered by users could help solving the situation, but we are investigating.

Sorry again for any inconvenience.

VPC connectivity issues are now stabilized.

Unlike stated previously, only Serverless Containers product is impacted.

We are now monitoring the situation.

Thanks for your understanding.

Incident is resolved now.

Impacts for users were limited to nl-ams region, and Serverless Containers product:

• From 10:00 UTC to 11:05 UTC: users received HTTP 5xx errors while calling their containers
• From 10:00 UTC to 12:00 UTC: some containers were stuck in creating/updating
• From 09:20 UTC to 12:20 UTC: some containers with VPC support lost connectivity to resources in their PN

These were all unexpected impacts of the following maintenance: https://status.scaleway.com/incidents/f7lpf0j1c2sq. The upgrade was partially applied, since we stopped it to stabilize the situation. While the situation is stable now, some work will still need to be done to fully complete the maintenance in the future, but we will postpone it for now.

In more details: a faulty step during our maintenance caused 1) our gateways to misbehave and start returning HTTP 503 errors and 2) VPC attached containers to be detached from their PN. The situation quickly snowballed, as a lot of pressure was made on the infrastructure, causing other unrelated issues (e.g. containers stuck in creating/updating statuses).

Despite our care, and successful upgrades on other regions (e.g. https://status.scaleway.com/incidents/tf28nvnckksz), the issue arose there because of nl-ams region being more crowded. We will perform additional tests before continuing the maintenance.

We deeply apologize for the inconvenience caused.