Skip to content
OutageDeck

Incident detail

K8s Customer Billing Data will be outdated

Resolved incidentMinor

Timeline window

to

Outage alerts

Get alerted the next time Harness breaks

Free email alerts for up to 5 providers — no card, live in about a minute. Paid plans add Slack, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Set up free alertsHow Harness alerts work

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Investigating

    We are currently experiencing an issue, K8s customer billing data will be stale.

  2. Resolved

    This incident has been resolved.

  3. Resolved

    ## Summary

    On May 4, 2026, after a service deployment in Prod2, background schema migrations did not complete successfully. As a result, some cluster and perspective-related data appeared missing or stale for Elevance and a small number of other Prod2 customers.

    The service itself remained available, but the database schema was not updated to match the new application code. This caused downstream billing and cluster data processing jobs to fail or skip expected data updates.

    ## Impact

    • Affected customers saw missing or stale cluster and perspective data in the UI.
    • Data appeared to stop updating around April 30 for impacted accounts.
    • Elevance was the primary impacted customer, along with a few other Prod2 customers.
    • There was no full service downtime.
    • Once remediated, the missing data was backfilled.

    ## Root Cause

    During the Prod2 deployment , the background schema migration process attempted to acquire a Redis distributed lock before running Timescale database migrations.

    The lock acquisition failed immediately due to a `PersistentLockException`, likely because another replica or overlapping deployment process was holding the lock. Since the migration used a zero-wait lock acquisition path, it did not retry and the migration did not run.

    The failure was logged only as a generic warning rather than a clear production error. Because of this, the migration failure was not immediately surfaced through alerts, and the service continued running with the database schema behind the application code expectations.

    ## Remediation

    The service was redeployed in Prod2. On restart, the background migrations completed successfully. The team then reran the required job to backfill the missing data and restore expected cluster and perspective data visibility.

    ## Preventive Actions

    • Update background migration locking behavior to wait/retry when acquiring the migration lock.
    • Improve logging from generic warnings to explicit error logs when schema migrations are skipped.
    • Add alerting for failed or skipped production schema migrations.