Skip to content
OutageDeck

Incident detail

FME - Some customers are experiencing delays in scheduled exports of impressions

Resolved incidentMinor1 affected service

Timeline window

to

Outage alerts

Get alerted the next time Harness breaks

Free email alerts for up to 5 providers — no card, live in about a minute. Paid plans add Slack, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Set up free alertsHow Harness alerts work

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Monitoring

    A fix has been implemented and we are monitoring the results.

  2. Monitoring

    We are seeing the backlogged items worked through and continuing to monitor results.

  3. Monitoring

    We are continuing to monitor results while the backlog is worked through.

  4. Resolved

    This incident has been resolved.

  5. Resolved

    Incident Summary

    Harness FME (Feature Management & Experimentation) faced substantial delays in scheduled impressions data exports on Jul 2, 2026. This issue stemmed from performance degradation within Tinybird's infrastructure, affecting query execution crucial to our export pipeline.

    Root Cause

    The core issue was a temporary performance degradation on the provider's infrastructure. This directly impacted the metadata retrieval endpoint, which failed to return job IDs intermittently. Consequently, completed export jobs underwent repeated retries, leading to a backlog in the export queue.

    Impact

    • The export backlog accumulated, peaking at more than 6 hours behind the planned schedule.
    • No data loss was detected during the incident.

    ‌

    Mitigation

    1. Adjusted Temporal workflows: Extended timeouts from ~30 to ~50 minutes and increased polling attempts from ~180 to ~300.

    2. Expanded resource capacity.

    3. Traffic redirection: Systematically shifted export workloads over to an optimized in-house cluster.

    4. Manual intervention: Terminated stalling workflows stuck in retry loops from failed metadata lookups.

    ‌

    Next Steps

    Complete the migration of all remaining export jobs to the internal analytics cluster (In progress).