[fr-par-1] - Issue with Compute Instances

Following a crash of one node on the block storage cluster at 20h15 UTC, some VMs are stuck.

The oncall team is investigating it.

We are continuing to investigate this issue.

We are still investigating this critical issue with the utmost priority.

Our team is currently working on solving the issue.

Disk I/O operations are starting to recover, which should begin resolving the issue for the instances.

The situation is stable.

All products are now operational except for the Kapsule public API, which is still unavailable.

The Kapsule public API is available again.

We are currently investigating this issue.

The situation has returned to normal

This incident has been resolved.

Incident Overview

On July 21, a block storage cluster incident affected the Instance product on FR-PAR-1 between 20:14 and 03:45 UTC. This incident caused a regional service degradation (FR-PAR), resulting in the following impacts:

• Kapsule operations were unavailable on a regional level between 21:30 and 01:30 UTC.
• Instances on FR-PAR-1 were non-functional.
• All managed products relying on Instance and Kapsule experienced disruptions.

#### Root Cause Analysis

5 whys

• Why were Instance and its dependent products unavailable?

Because disk I/O was impossible.

• Why was disk I/O impossible?

Because cluster doesn't accept I/O operations.

• Why cluster doesn’t accept I/O operations?

Due to a fault in a cluster component (crash).

• Why was a cluster component faulty?

It suffered an OOM (Out of Memory) kill.

• Why did an OOM kill occur?

The memory limit was incorrectly configured, causing the component to consume excessive memory and triggering an OS-level OOM kill.

• Why was the memory limit incorrectly configured?

The cluster hardware is heterogeneous; nodes have varying memory capacities. A global setting was applied without accounting for nodes with lower memory, leading to node-level OOM kills.

• Why was the configuration error not detected?

We lack sufficient safeguards for this specific configuration.

### Impact on Kapsule Product

‌

• Why did an incident in a single Availability Zone (AZ) impact the entire FR-PAR region for Kapsule?The Kapsule API was proactively disabled to prevent a cascading failure ("snowball effect") caused by auto-healing mechanisms reacting to widespread instance unavailability.
• Why is a multi-AZ product affected by a single API endpoint failure?The Kapsule API architecture is currently regional and lacks the granularity to isolate or disable specific Availability Zones. Consequently, disabling the regional API was the necessary precautionary measure to protect block storage convergence.

#### Summary of Events

#### Incident Timeline (UTC)

| Time (UTC) | Event Description |

| --- | --- |

| 20:14 UTC | OOM kill on one OSD |

| 20:28 UTC | First alert on block storage team |

| 20:43 UTC | Escalation to larger teams, incident open at company level |

| 20:48 UTC | sbs-api stops processing river jobs (no more volume update) |

| 21:30 UTC | API Kapsule is unavailable |

| 21:43 UTC | First restart of blk-api (internal api, not customer facing) |

| 22:27 UTC | Second restart of blk-api, helped unstick api calls |

| 23:48 UTC | Faulty OSD removed from production, throughput restored on cluster |

| 23:50 UTC | Teams begin relaunching operations on disk |

| 00:00 UTC | System was unavailable |

| 01:30 UTC | API Kapsule is available |

| 02:08 UTC | Beginning of second block cluster global failure |

| 03:45 UTC | Block Cluster state restored successfully, end of impact |

#### Resolution and Improvements

Short-term Actions

• During the incident timelapse, we identified a configuration issue and corrected the memory limits across all cluster nodes.

Mid-term Actions

• Enhance cluster configuration management to prevent environment-specific mismatches (e.g., node-level hardware heterogeneity).
• Implement isolation mechanisms to ensure that individual component failures do not impact the overall cluster stability.
• We also notice network saturation during the recovery leading to latencies increase, we may need to rework this part.
• Optimize network performance during recovery phases to prevent latency spikes caused by saturation.
• Improve the incident response to avoid logical bias

#### Contact

If you have any further questions or need assistance, please contact our support team.