Skip to content

Incident detail

FME SDKs - SDKs falling back to polling

Resolved incidentMinor

Timeline window

to

Outage alerts

Get alerted the next time Harness breaks

Free email alerts for up to 5 providers — no card, live in about a minute. Paid plans add Slack, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Identified

    Instability in streaming connections could cause a delay in propagating updates. All SDKs are designed to poll against the CDN on streaming disconnects and instability scenarios.

    This could also cause some noise in the logs as SDKs try to reconnect to streaming. But updates are unaffected.

  2. Monitoring

    A fix has been implemented and we are monitoring the results.

  3. Resolved

    This incident has been resolved.

  4. Resolved

    ### Summary

    On June 4, 2026, customers using Harness FME SDKs experienced degraded streaming performance. SDK connections repeatedly disconnected and fell back to polling mode, resulting in delayed flag propagation and elevated log noise. Flag evaluation itself was unaffected — all SDKs are designed to poll against the CDN during streaming disconnects.

    ### Impact

    • Region affected: us-east-1 (other regions — EU, APAC — remained healthy)
    • Customer impact: SDK streaming connections intermittently dropped, causing SDKs to fall back to polling. Flag evaluation continued uninterrupted; customers experienced delayed flag propagation

    ### Root Cause

    The underlying cause was memory exhaustion in the backend messaging cluster in the us-east-1 region, triggered by a combination of an ongoing service rollout and a traffic spike during the same . Other regions remained fully healthy throughout.

    ### Remediation Actions Taken

    1. Service was rolled back to the previously stable version.

    2. Increased capacity to handle the surge of connections.

    ### Preventive Measures / Next Steps

    To prevent such issues happening again, Harness is working on several optimizations:

    • Enhance deployments to do validation before broader rollout to catch capacity pressure early.
    • Improve connection handling during deployments or more conservative deployment pacing during high-load periods.
    • Improved SDK behavior on reconnect: Connection recycles inherently generate a resync storm; further investigation ongoing into reducing the cascading request impact from mass reconnects