Monitoring - Pipelines Stuck - Prod2

We are monitoring the stuck pipelines in prod2. The new executions are passing as we are continuously monitoring the services.

We are monitoring the stuck pipelines in prod2. For the customers who are still seeing stuck pipelines, we request you to abort and re-trigger.

This incident has been resolved.

## Summary

On August 6, 2026 (morning PDT), some customers running pipelines in the Prod2 production environment observed pipeline executions that stopped making progress — stages that did not advance and produced no further output or status updates. The issue was reported by affected customers. Harness engineers identified the cause, mitigated the impact, and pipeline executions returned to normal operation.

The issue was caused by a self-referential pipeline expression. A Git webhook triggered a pipeline that referenced the contents of the webhook payload, and the payload itself contained further copies of that same expression. Each round of expression resolution therefore produced more expressions to resolve, doubling the amount of work each time. This exhausted the resources of the service instance processing that execution, and other executions assigned to the same instance were unable to progress while it was in that state.

## Impact

During the incident window (approximately 6:11 AM to 11:23 AM PDT on August 6, 2026):

• Some customers' pipeline executions on Prod2 stalled mid-execution and made no further progress.
• Affected executions produced no new step output or status updates, and had to be aborted and re-run after mitigation.
• Behavior was limited to executions being processed by the affected service instance — pipelines handled by other instances continued to execute normally.

There was no data loss. Pipeline definitions, execution history, and stored state were unaffected. The majority of pipelines on Prod2 continued to execute successfully throughout the incident; the primary impact was that some in-flight executions could not complete and needed to be re-run once the issue was mitigated.

## Root Cause

Harness pipelines support expressions that are resolved at runtime — for example, an expression that inserts the contents of the Git webhook payload that triggered the pipeline.

In this case, a Git commit message contained the literal text of the payload expression itself, twice, and the pipeline referenced that same payload expression. Because the commit message is part of the webhook payload, resolving the expression inserted the entire payload — including the two literal copies of the expression carried in the commit message. Those newly inserted copies were then treated as expressions to be resolved, and each pass inserted two more full copies of the payload. The size of the value being processed, and the work required to process it, therefore doubled on every pass and grew exponentially rather than converging.

Harness has a safeguard intended to stop exactly this: expression resolution is bounded by a maximum nesting depth, beyond which resolution halts and the pipeline fails with an explicit error. A defect in that safeguard meant the limit was not applied in this specific self-referential case, so resolution continued unchecked.

Expression resolution runs inline on the threads that start pipeline steps. As each pass consumed progressively more memory and CPU without ever completing, the service instance performing that work stopped making progress, and every execution assigned to that instance stalled — which is what customers reported.

## Mitigation

Harness completed the following immediate mitigation steps:

• Identified the pipeline and the expression pattern responsible for the runaway resolution.
• Stopped the affected service instance so that it would take on no further work. The remaining healthy instances picked up and processed queued executions normally.
• Confirmed that pipeline executions returned to normal and closed the incident.

These actions restored normal pipeline execution behavior and resolved the customer-facing impact.

## Action Items

To reduce the risk of recurrence and improve detection, the following actions are in various stages of being implemented:

• Fix the defect in the expression depth and loop-detection safeguard so that self-referential expressions are caught and fail fast with a clear error instead of consuming resources without bound.
• Prevent payload expressions from being resolved out of trigger payload content, removing the self-referential path entirely.
• Tighten the maximum expression nesting depth and evaluate explicit loop detection in addition to the existing depth limit.
• Enhance automated tests in pre-production environments that reproduce self-referential expression patterns and verify that the safeguard detects and stops them.
• Add monitoring for this pattern in pipeline executions so that it is detected proactively.