Intermittent External Network Connectivity Issues Affecting Build VMs

Summary - We are intermittently facing network connectivity issues with our Build VM's unable to connect to external resources.

We are currently investigating the issue.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

## Summary

Starting on August 4, 2026, CI runners in the us-west1 and us-central1 regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services such as GitHub and Bitbucket over outbound network gateways.

## Impact

• CI runners in the affected regions intermittently experienced connection timeouts of approximately 134 seconds when reaching external services (e.g., GitHub, Bitbucket) over our outbound network gateways.
• The issue was intermittent rather than constant — connections succeeded under normal load, and failures clustered during periods of high outbound traffic volume.
• No data was lost or corrupted. This was a network-connectivity and capacity issue, not a data-integrity issue.
• us-west1 and us-central1 were the affected regions; other regions were not impacted by this issue.

## Root Cause

‌

Our load balancer distributes outbound traffic across multiple NAT gateways using a hashing method based on connection details (source/destination address and port). For any single connection, these details stay constant for that connection's lifetime. We had a sustainted traffic surge for a few seconds which congested the gateways

‌

## Action Items

To prevent such issues from happening again Harness will,

Increase outbound connection capacity on our NAT gateways by provisioning additional external network interfaces, giving each gateway a substantially larger pool of connections it can serve concurrently..