GitHub and Other VCS workloads experiencing intermittent issues

[GitHub.com](http://GitHub.com "GitHub.com") has acknowledged an issue causing intermittent issues accessing their API. This may cause VCS-based Terraform workloads to malfunction. We are monitoring GitHub status and will update this issue as we learn more.

Current impact state: VCS workloads experiencing intermittent issues.

Action underway: Awaiting word from [GitHub.com](http://GitHub.com "GitHub.com")

Next update time: 17:30 UTC

We are continuing to receive reports of intermittent issues with VCS workloads, some of which are not hosted on [GitHub.com](http://GitHub.com "GitHub.com"). Our engineering team is investigating these issues.

Current impact state: VCS workloads experiencing intermittent issues.

Action underway: Awaiting word from [GitHub.com](http://GitHub.com "GitHub.com")

Next update time: 18:00 UTC

Our engineering is continuing to investigate intermittent VCS-related failures.

Current impact state: VCS related workloads experiencing intermittent issues.

Action underway: Engineering team investigating.

Next update time: 18:30 UTC

Our engineers are no longer able to reproduce the issue and error counts have dropped off. We will continue to monitor for any further issues. If you experience any further problems please reach out to support.

Current impact state: System in recovery.

Action underway: Monitoring.

Next update time: 19:00 UTC

Intermittent and some persistent VCS issues have been reproduced on some workspaces. Our engineering team is continuing to investigate.

Current impact state: Intermittent issues on VCS workspaces are continuing to be observed.

Action underway: Engineering team is working on reliably reproducing the issue.

Next update time: 19:00 UTC

Our engineering team has identified the tfc-agent release v1.33.2 as the cause of intermittent issues with VCS providers. We have downgraded the version to 1.33.1 for remote runs. If you are experiencing issues with self-hosted agents please downgrade to 1.33.1 until a new release is made to address the issue.

Current impact state: Remote runs fully-functional. Affected self-hosted agents need to downgrade to tfc-agent version 1.33.1. Stacks users may experience a regression when using proxies with version 1.33.1.

Action underway: Releasing a fixed tfc-agent version.

Next update time: 19:45 UTC

Downgrading tfc-agent to 1.33.1 has proven effective in eliminating the intermittent VCS failures. We will continue to monitor for any further impacts. A new release with a fix is in-progress and expected within 24 hours. If you are not affected, do not downgrade. If you experience further issues please contact support.

VCS connection issues should be mitigated by downgrading to tfc-agent 1.33.2. Our engineering team is still investigating the cause of this failure. If you do not have any VCS connection problems you do not need to downgrade to 1.33.1. We will release a new version of tfc-agent in the next 24 hours to fix the root cause.

One minor correction, VCS connection issues can be mitigated by downgrading tfc-agent from 1.33.2 to 1.33.1.

Today we released tfc-agent 1.33.3 to address the root cause of the failures we saw yesterday. A recent security patch to libcurl exposed a bug in the version of git we ship in the tfc-agent image. This causes some VCS setups to fail when the VCS provider issues an HTTP redirect. We've patched the bug in git. If you are using remote runs, you don't have to do anything, you're already on 1.33.3. If you downgraded to 1.33.1 to work around git-based VCS problems, you can safely upgrade to 1.33.3 now. Please contact support if you experience any further issues.