Skip to content
OutageDeck

Incident detail

Feature Flags unable to update

Resolved incidentMinor1 affected service

Timeline window

to

Outage alerts

Get alerted the next time Harness breaks

Free email alerts for up to 5 providers — no card, live in about a minute. Paid plans add Slack, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Set up free alertsHow Harness alerts work

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Investigating

    We are currently investigating this issue.

  2. Investigating

    We are continuing to investigate this issue.

  3. Monitoring

    A fix has been implemented and we are monitoring the results.

  4. Resolved

    This incident has been resolved.

  5. Resolved

    ## Summary

    On April 15, 2026, between approximately 23:21 UTC and 01:58 UTC, customers using Feature Flag in the prod2 environment experienced delays in feature flag updates.

    Feature flag changes made via UI or API were successfully processed but were not immediately reflected, causing stale flag values to be served.

    ## Impact

    • Scope: Customers on prod2 environment only
    • Customer Impact:
    • Feature flag updates were delayed or appeared ineffective
    • Applications continued serving stale configurations
    • Other Environments: No impact to prod0, prod1, or other regions

    ## Root Cause

    The issue was caused by replication lag in the read replica database used for serving feature flag reads. A long-running read query on the replica blocked replication updates from the primary database. This caused a delay in propagating recent feature flag changes to read queries

    ### What triggered the issue

    • A high-volume API usage pattern involving large paginated queries on target data. These queries became resource-intensive impacting the database.

    ## Mitigation

    ### Immediate Actions

    • Identified and terminated long-running queries on the replica
    • Replication resumed and flag updates began reflecting correctly

    ## Prevention & Next Steps

    We are continuing to strengthen reliability through:

    • We configured replica to automatically cancel queries that block replication beyond a threshold and tuned query timeouts for heavy read operations
    • Improving query efficiency and pagination strategies
    • Enhancing monitoring and alerting for replication lag
    • Evaluating database upgrades and scaling improvements