Errors Opening Repositories

We're investigating reports of customers unable to open projects, receiving error 500 instead.

We're investigating reports of customers unable to open projects, receiving error 500 instead. Details in https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22925.

No material updates are available at this time. Our team continues to investigate the root cause of the issue.

No material updates are available at this time. Our team continues to investigate the root cause of the issue.

We’ve identified an abuse vector as the root cause and begun mitigation efforts. Early signs indicate that the service is recovering.

We’ve identified an abuse vector as the root cause and begun mitigation efforts. Early signs indicate that the service is recovering.

We’re continuing mitigation efforts and have observed some recovery. Customers may still encounter HTTP 500 errors in the UI and during Git operations when accessing certain projects.

We’re continuing mitigation efforts and have observed some recovery. Customers may still encounter HTTP 500 errors in the UI and during Git operations when accessing certain projects.

We are continuing mitigation and monitoring recovery. Customers may still encounter HTTP 500 errors in the UI and during Git operations for certain projects. We will provide a further update as mitigation progresses.

We are continuing mitigation and monitoring recovery. Customers may still encounter HTTP 500 errors in the UI and during Git operations for certain projects. We will provide a further update as mitigation progresses.

Mitigation has not yet fully restored the GitLab UI and Git operations in certain projects. We are still investigating HTTP 500 errors and will provide a further update as more information becomes available.

Mitigation has not yet fully restored the GitLab UI and Git operations in certain projects. We are still investigating HTTP 500 errors and will provide a further update as more information becomes available.

Investigation continues. We will shortly be restarting the affected Gitaly node after reconfiguration. Further updates will be provided accordingly.

Investigation continues. We will shortly be restarting the affected Gitaly node after reconfiguration. Further updates will be provided accordingly.

The restart of the affected Gitaly node did not improve the issue; HTTP 500 errors continue affecting the GitLab UI and Git operations in certain projects. We are investigating and will provide further updates as the situation changes.

The restart of the affected Gitaly node did not improve the issue; HTTP 500 errors continue affecting the GitLab UI and Git operations in certain projects. We are investigating and will provide further updates as the situation changes.

Further mitigation is being rolled out while we continue to investigate HTTP 500 errors affecting the GitLab UI and Git operations. We will provide further updates as the mitigation progresses.

Further mitigation is being rolled out while we continue to investigate HTTP 500 errors affecting the GitLab UI and Git operations. We will provide further updates as the mitigation progresses.

A further mitigation change has been applied, and we are assessing its impact. HTTP 500 errors continue affecting the GitLab UI and Git operations in certain projects. Further updates will follow.

A further mitigation change has been applied, and we are assessing its impact. HTTP 500 errors continue affecting the GitLab UI and Git operations in certain projects. Further updates will follow.

Further mitigation change are currently being applied, and we are continuing to investigate the issue. HTTP 500 errors may continue affecting the GitLab UI and Git operations in certain projects. Further updates will follow.

Further mitigation change are currently being applied, and we are continuing to investigate the issue. HTTP 500 errors may continue affecting the GitLab UI and Git operations in certain projects. Further updates will follow.

We are currently rolling back the deployment of Gitaly. We're waiting for the deploy to land, and we are continuing the investigation.

We are currently rolling back the deployment of Gitaly. We're waiting for the deploy to land, and we are continuing the investigation.

A rollback release has been deployed to all Gitaly nodes. They are stable at present. However, we are continuing to monitor the situation.

A rollback release has been deployed to all Gitaly nodes. They are stable at present. However, we are continuing to monitor the situation.

Gitaly nodes have stabilized following the completed rollback deployment. The incident has been mitigated and we remain focused on continued monitoring.

Gitaly nodes have stabilized following the completed rollback deployment. The incident has been mitigated and we remain focused on continued monitoring.

The incident has been resolved following the completion of the rollback deployment across the Gitaly nodes. All nodes have returned to a stable state, repository access has been restored. Incident team is currently investigating the root cause.