Provider
IncortaIncident detail
Experiencing networking issues in GCP regions
Timeline window
to
Get alerted the next time Incorta breaks
Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.
Timeline
Incident updates
Every update on the official status source, oldest to newest, exactly as it appeared there.
Investigating
We are currently investigating the issue
Identified
The issue has been identified and fix implementation is in progress
Issue: Forced upgrade for network data plane leads to incompatible versions between master and workers
Identified
mitigation is in progress for GCP - Europe West
Identified
We are continuing to work on a fix for this issue.
Identified
We are continuing to work on a fix for this issue.
Monitoring
Mitigation applied and currently monitoring network data plane
Resolved
The incident has be resolved.
Incident Report:
### Summary:
On 17 February , Network Data plane Takes Longer than Usual To Initialize.
Customers in multiple GCP domains (cloud3, cloud4, cloud5 and cloud6) suffers from spark workloads pending for healthy compute nodes longer than usual.
### Root Cause:
Forced upgrade for network data plane leads to degraded performance of network data plane of workers.
The aim of the forced upgrade was to provide better secure and performance for management data planes.
The forced upgrades was tested in internal different regions, but due to high concurrency and requests in cloud4, cloud5 and cloud6 it took some time to stabilize network data plane.
### Remediation and Prevention:
Incorta Cloud Engineers alerted with workloads takes longer than usual, started immediately to investigate and mitigate the. issue within an hour of issue discovery.
High demand of spark executors and slowness of spark executors to starts was correlated with the forced upgrade event of the data plane.
If your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you. Incorta Cloud is committed to preventing a repeat of this issue in the future and is completing the following actions:
[Mitigation] Upgrade deprecated compute workloads. [Done]
[Prevent] Exclude GKE clusters from the forced upgrade window of cloud provider. [Done]
[Monitor] Improve monitoring of spark workload failures in Arcane. [In progress]
Keep exploring
More from Incorta
Neighboring incidents on Incorta's timeline and the rest of their record on OutageDeck.