CI Linux Cloud VM Networking issues - intermittent

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

### Summary

Between approximately 11:37 AM and 3:01 PM PDT (18:37–22:01 UTC) on September 29, 2026, customers using Harness CI self-hosted Linux cloud runners in prod1/prod2 (us-west1) experienced intermittent networking failures during pipeline execution — primarily Git/network timeouts when fetching source code.

Root Cause

This was primarily due to saturation of our NAT which was futher exacerbated by a temporary cloud provider compute capacity, preventing new VM provisioning there.

### Impact

• Customers running self-hosted Linux CI builds in us-west1 saw intermittent build/pipeline failures, not a full outage.
• Observed customer-facing errors included Git fetch timeouts, e.g. `Failed to connect to github.com port 443: Connection timed out`

Remediation

• Immediate: Rebalanced ARM build-VM load from the saturated us-west1 path toward central1 (partially limited by central1 capacity); restarted the affected Cloud NAT gateways, which stabilized pipeline networking.

Action Items

1. Secure additional NAT gateway/IP capacity (In progress)

2. Improve NAT saturation detection and alerting ahead of customer impact.

3. Evaluate cross-region capacity contingencies so a single region's compute shortage doesn't block failover mitigation.