Skip to content

Incident detail

Degraded Performance — Feature Flags in PROD2

Resolved incidentMinor1 affected service

Timeline window

to

Get alerted the next time Harness breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update Harness posted, oldest to newest, exactly as it appeared on their official status page.

  1. Investigating

    We are currently investigating this issue.

  2. Monitoring

    A fix has been implemented and we are monitoring the results.

  3. Resolved

    This incident has been resolved.

  4. Resolved

    ### Summary

    On April 24, 2026, a large non-batched bulk DELETE operation on the prod-2 primary database triggered lock contention, causing Feature Flag API latency and hung queries across multiple customer SDKs.

    ### Impact

    1. Slow SDK auth/init — SDKs took longer than expected to complete evaluations

    2. Elevated latency across many FF APIs

    3. Limited to Feature Flag module, prod-2

    4. No Data loss

    ### Root Cause

    A background cleanup job executed a non-batched, single-transaction delete causing lock contention and API latency spikes

    Mitigation

    Immediately terminated the offending queries.

    ### Next Steps / Action Items

    To prevent such issues from happening again. we are working on

    1. Enhanced alerting and observability on long running queries.

    2. Permanently replace large single-transaction delete pattern with smaller batched deletes

Keep exploring

More from Harness

Neighboring incidents on Harness's timeline and the rest of their record on OutageDeck.