Skip to content
OutageDeck

Incident detail

[COCKPIT] - [fr-par/pl-waw] - Scaleway Metrics/Logs ingestion down

[COCKPIT] - [fr-par/pl-waw] - Scaleway Metrics/Logs ingestion down

Resolved incidentMinor0 affected services

Timeline window

to

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Monitoring

    Cockpit impacted this night betwen 1am30 and 8am30 with data loss for all Products metrics and logs on FR-PAR & PL-WAW Regions Root cause : based on last week Cockpit ingestion solution, a high RAM consumption saturated nodes and corrupted replay solution. Status : Cockpit fixed at 8am30 and fix in development to avoid replay corruption on OOM events

  2. Monitoring

    Root cause was finally identified yesterday evening, we make an evolution on staging environments and monitored its behavior today. We plan to deploy this fix next monday morning. Metrics & Logs acquisition are fine since yesterday with a scaling workaround, we keep this solution for the week-end to not impact production on a friday afternoon.

  3. Monitoring

    We applied and monitored today the last fix for high RAM consumption. All seems fine now on Cockpit performances for all regions.

  4. Resolved

    This incident has been resolved.