Skip to content

Incident detail

Experiencing networking issues in GCP regions

Resolved incidentMinor1 affected service

Timeline window

to

Get alerted the next time Incorta breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Investigating

    We are currently investigating the issue

  2. Identified

    The issue has been identified and fix implementation is in progress

    Issue: Forced upgrade for network data plane leads to incompatible versions between master and workers

  3. Identified

    mitigation is in progress for GCP - Europe West

  4. Identified

    We are continuing to work on a fix for this issue.

  5. Identified

    We are continuing to work on a fix for this issue.

  6. Monitoring

    Mitigation applied and currently monitoring network data plane

  7. Resolved

    The incident has be resolved.

    Incident Report:

    ### Summary:

    On 17 February , Network Data plane Takes Longer than Usual To Initialize.

    Customers in multiple GCP domains (cloud3, cloud4, cloud5 and cloud6) suffers from spark workloads pending for healthy compute nodes longer than usual.

    ### Root Cause:

    Forced upgrade for network data plane leads to degraded performance of network data plane of workers.

    The aim of the forced upgrade was to provide better secure and performance for management data planes.

    The forced upgrades was tested in internal different regions, but due to high concurrency and requests in cloud4, cloud5 and cloud6 it took some time to stabilize network data plane.

    ### Remediation and Prevention:

    Incorta Cloud Engineers alerted with workloads takes longer than usual, started immediately to investigate and mitigate the. issue within an hour of issue discovery.

    High demand of spark executors and slowness of spark executors to starts was correlated with the forced upgrade event of the data plane.

    If your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you. Incorta Cloud is committed to preventing a repeat of this issue in the future and is completing the following actions:

    [Mitigation] Upgrade deprecated compute workloads. [Done]

    [Prevent] Exclude GKE clusters from the forced upgrade window of cloud provider. [Done]

    [Monitor] Improve monitoring of spark workload failures in Arcane. [In progress]

Keep exploring

More from Incorta

Neighboring incidents on Incorta's timeline and the rest of their record on OutageDeck.