Skip to content
OutageDeck

Incident detail

Delayed Metrics

Delayed Metrics

Resolved incidentMajor0 affected services

Timeline window

to

Sponsored placement

Google AdSense placeholder

This slot is reserved for future AdSense placement. It stays as a non-invasive placeholder until a client ID and slot ID are configured.

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Investigating

    We are investigating issues with customer facing metrics in the fly-metrics.net dashboard. Users may see delayed or missing metrics at this time.

  2. Identified

    We've identified an issue on multiple hosts causing delayed metric ingestion into our hosted fly-metrics.net dashboards. We are working on a fix.

  3. Identified

    We are continuing to see delayed metric exports from a number of hosts. Users will see delayed or missing metrics for machines on impacted hosts at this time.

  4. Identified

    We are continuing to see delayed metrics due to resource contention on a subset of metrics ingestion hosts, and we are working to rebalance ingestion traffic and reduce the backlog.

  5. Identified

    We are still in the process of increasing the metrics cluster throughput to catch up with metric backlog.

  6. Identified

    Metric ingestion is still delayed for some customers. We’re seeing gradual improvement and continue to monitor the situation while the system catches up.

  7. Identified

    Backlogged metrics data is still being processed, continue to monitor the situation while the system catches up.

  8. Identified

    Backlogged metrics are still being processed and ingestion delays persist for some customers, but we’re continuing to see gradual improvement as the backlog continues to drain.

  9. Identified

    Backlogged metrics are still being processed and ingestion delays persist for some customers

  10. Identified

    Backlogged metrics are still being processed. We're bringing extra processing capacity online to speed up the process.

  11. Identified

    Almost all metrics have caught up aside from a small handful in `sin` and `syd`. We're continuing to monitor and expect these to complete in the next few hours.

  12. Resolved

    This incident has been resolved.