{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Teams, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-10-12T02:49:54.210Z"
  },
  "data": {
    "id": "incident_statuspage_mixpanel_k0f61r4zfp44",
    "slug": "mixpanel-query-api-degraded-performance-2026-05-14",
    "title": "Query API degraded performance",
    "summary": "Query API degraded performance",
    "status": "resolved",
    "stale": false,
    "severity": "minor",
    "startedAt": "2026-05-14T23:17:21+00:00",
    "updatedAt": "2026-05-21T22:03:05.083+00:00",
    "resolvedAt": "2026-05-16T02:50:50.66+00:00",
    "provider": {
      "slug": "mixpanel",
      "name": "Mixpanel"
    },
    "affectedServices": [
      {
        "slug": "mixpanel-application",
        "name": "Application"
      }
    ],
    "links": {
      "html": "/incidents/mixpanel-query-api-degraded-performance-2026-05-14",
      "api": "/api/v1/incidents/mixpanel-query-api-degraded-performance-2026-05-14",
      "providerHtml": "/providers/mixpanel",
      "alerts": "https://outagedeck.com/account?stack=mixpanel&utm_source=api&utm_medium=response&utm_campaign=api_alerts&utm_content=incident"
    },
    "impactSummary": "Mixpanel reported a minor event for the affected tracked services.",
    "source": {
      "id": "source_mixpanel_status",
      "kind": "official_api",
      "name": "Mixpanel Status",
      "checkedAt": "2026-10-12T02:40:10.21+00:00",
      "stale": false,
      "officialUrl": "https://www.mixpanelstatus.com",
      "statusPageUrl": "https://www.mixpanelstatus.com"
    },
    "updates": [
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_t4y9mjdc7cpz",
        "status": "investigating",
        "body": "We are experiencing degraded performance with our Query API, including increased latency. You may see slow-loading reports, incomplete query results, query errors, or data discrepancies. We appreciate your patience while our engineers work to restore normal functionality. We will post progress updates on our status page. If you have any questions, please contact support.",
        "createdAt": "2026-05-14T23:17:21.811+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_2yf7jx6dbwbb",
        "status": "identified",
        "body": "The issue has been identified and a fix is being implemented.",
        "createdAt": "2026-05-14T23:48:22.775+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_zl4npcf3cd0q",
        "status": "identified",
        "body": "We are continuing to work on a fix for this issue.",
        "createdAt": "2026-05-15T00:18:26.923+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_0g9dtmsm9r7x",
        "status": "identified",
        "body": "We are continuing to work on a fix for this issue.",
        "createdAt": "2026-05-15T00:49:36.222+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_yzcsds0ctnts",
        "status": "identified",
        "body": "We are continuing to work on a fix for this issue.",
        "createdAt": "2026-05-15T01:49:58.431+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_sgkjk39w48ns",
        "status": "identified",
        "body": "We are continuing to work on a fix for this issue.",
        "createdAt": "2026-05-15T02:49:55.547+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_l8q8lhbqv1dh",
        "status": "identified",
        "body": "We are continuing to work on a fix for this issue.",
        "createdAt": "2026-05-15T03:49:39.235+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_tsrsnw61xp5g",
        "status": "identified",
        "body": "We are continuing to work on a fix for this issue.",
        "createdAt": "2026-05-15T08:18:20.249+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_01ldqtjj10m3",
        "status": "identified",
        "body": "We are continuing to work on a fix for this issue.",
        "createdAt": "2026-05-15T10:19:26.022+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_3rf1nz4dqtxg",
        "status": "identified",
        "body": "We've continued to make progress on the issue affecting some US projects. Query success rates have returned to normal levels, and the related processing delays have also recovered.\nA subset of affected projects may still see incomplete data in query results while our recovery process runs. We're actively working on this and will share another update in 2 hours.\nImpact remains limited to our US region. We do not currently believe any data has been permanently lost.",
        "createdAt": "2026-05-15T12:28:31.734+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_75byngch3z3g",
        "status": "identified",
        "body": "Recovery is progressing well. Query success rates remain at normal levels, and missing data has now been restored for a portion of affected projects.\nWe're continuing the recovery process for the remaining affected projects and currently estimate full recovery within approximately 3–5 hours.\nSome customers may still see incomplete data in query results until this work is complete.",
        "createdAt": "2026-05-15T14:31:36.076+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_lwvm78dvd39y",
        "status": "identified",
        "body": "We are continuing to work on resolving this issue. Our current estimate for full recovery is approximately 3–5 hours. Some customers may still see incomplete data in reports until recovery is complete. We will continue to provide updates.",
        "createdAt": "2026-05-15T16:37:44.885+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_06441h6s6qky",
        "status": "identified",
        "body": "Recovery is continuing to progress and query latency has returned to normal levels. Our current estimate for full recovery is approximately 3–4 hours. Some customers may still see incomplete data in reports until recovery is complete. We will continue to provide updates.",
        "createdAt": "2026-05-15T18:47:57.059+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_qs0x5qzzhvxf",
        "status": "identified",
        "body": "We are investigating an increase in query latency that appears to be unrelated to the ongoing recovery. You may experience slower-loading reports. Our team is actively looking into the cause and we will provide an update shortly.\n\nRecovery is continuing to progress. Our current estimate for full recovery is approximately 3–4 hours. Some customers may still see incomplete data in reports until recovery is complete. We will continue to provide updates.",
        "createdAt": "2026-05-15T19:56:24.941+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_mvxjpmymwz4x",
        "status": "identified",
        "body": "Query latency has returned to normal levels. We are continuing to monitor.",
        "createdAt": "2026-05-15T21:23:24.187+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_8l9ks6qx916c",
        "status": "identified",
        "body": "Query latency has returned to normal levels and has been resolved.\n\nRecovery is continuing to progress, and our current estimate for full recovery is approximately 4–5 hours. Some customers may still see incomplete data in reports until recovery is complete. We will continue to provide updates.",
        "createdAt": "2026-05-15T22:14:51.231+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_llw6rpy2t1mm",
        "status": "identified",
        "body": "Recovery is continuing to progress. Our current estimate for full recovery is approximately 1-2 hours. Some customers may still see incomplete data in reports until recovery is complete. We will continue to provide updates.",
        "createdAt": "2026-05-16T00:17:59.195+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_9vjf1rt01g8q",
        "status": "monitoring",
        "body": "A fix has been implemented and we are monitoring the results.",
        "createdAt": "2026-05-16T01:52:29.746+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_tyfxrmt4r8hp",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2026-05-16T02:50:50.66+00:00"
      },
      {
        "id": "update_statuspage_mixpanel_k0f61r4zfp44_d0tlm35mh413",
        "status": "resolved",
        "body": "# Mixpanel RCA: Transient Data Access Issue, May 14, 2026\n\n## Summary\n\nOn Thursday, May 14, 2026 at approximately 2:30 PM PT, a routine but infrequent cleanup operation in Mixpanel's storage system mistakenly removed a portion of production data files in addition to the unused files it was intended to remove. Some customers experienced query errors during the hours that followed. We detected the issue within minutes, deployed mitigations the same evening that returned query success rates and latency to normal, and restored the affected files from backup by 5:15 PM PT on Friday, May 15. Mixpanel's ingestion pipeline was not affected and no event data was lost in transit.\n\n## What happened\n\nThis incident was triggered by a storage cleanup procedure that runs periodically to remove files no longer referenced by Mixpanel's metadata. The procedure was more involved than usual: it followed a recent enhancement to our file storage strategy that left a set of unused files behind in our storage backend, and addressing them required extending our standard cleanup approach to cover a new code path.\n\nAs part of executing this extended cleanup, an engineer generated the list of files to delete using a SQL query whose date filter was not strictly earlier than the reference snapshot it was being compared against. As a result, a small set of legitimate production files that had been written in the gap window between the snapshot and the filter date were incorrectly classified as unused and removed.\n\nThe deletion ran for roughly half an hour before internal alerting caught the resulting query failures and the operation was stopped. The trigger was operator error against an ambiguous runbook, not a defect in the live serving path or in our ingestion pipeline.\n\n## Customer impact\n\nImpact unfolded in two phases.\n\nThe first phase ran from Thursday at approximately 2:30 PM PT until 8:11 PM PT — roughly five and a half hours. During this window, customers across the platform may have seen slower or failed queries when their requests touched files that had been deleted. The breadth and severity varied by project depending on which data each query touched. By 8:11 PM PT, mitigations had fully rolled out — queries automatically retried against an alternate availability zone, and a fallback path was put in place to serve missing files from a backup datastore. After this point, query success rate and latency returned to normal.\n\nThe second phase lasted from 8:11 PM PT Thursday through approximately 5:15 PM PT Friday, May 15. During this window, fewer than 2% of customers were still affected — specifically, those whose deleted files had not yet been fully restored from backup. The vast majority of these files were recovered by Friday afternoon. A small number of projects \\(under 30\\) had files that could not be fully recovered from backup, and we are following up with those accounts directly.\n\n## Timeline \\(Pacific Time\\)\n\n* May 14, 2:30 PM — Cleanup operation begins\n* May 14, 3:11 PM — Internal alerting flags query failures; the cleanup operation is stopped within minutes\n* May 14, 4:07 PM — Status page banner posted\n* May 14, 4:45 PM — Mitigation deployed: queries automatically retry against an alternate availability zone\n* May 14, 7:12 PM — Mitigation deployed: queries fall back to a backup datastore for missing files\n* May 14, 8:11 PM — Query success rate and latency fully restored to normal levels\n* May 15, 5:15 PM — File restore from backup complete; status page banner resolved\n\n## Why this happened\n\nSeveral contributing factors lined up.\n\nThe runbook for this cleanup procedure had ambiguous wording around the ordering and timing of its inputs. It had been recently authored to handle the new file-storage code path and had not gone through a formal review before being used.\n\nOur cleanup tooling did not programmatically enforce the safety invariant that the date filter must be strictly before the reference snapshot. That invariant lived only in operator-authored SQL.\n\nThe extended cleanup was being executed in parallel across two storage layers by two different engineers, which increased the room for error.\n\n## What we're doing to prevent recurrence\n\nWe have already made or have actively in flight the following changes.\n\nWe are adding programmatic safeguards to our cleanup tooling so that an input set whose date filter is not safely before the reference snapshot is rejected before any deletion occurs, along with a reconciliation step that flags any production-referenced file before deletion proceeds.\n\nDestructive cleanup operations will now run in phased stages, starting with internal projects and pausing for a holding period before any broader execution.\n\nDestructive storage operations now require a second engineer to sign off on the exact deletion set and to be present during execution, matching the practice we already follow for database migrations.\n\nWe have updated the cleanup runbook with explicit guidance on input timing, required safety buffers, and an enforced review process for any runbook covering a destructive operation.\n\nLonger term, we are working to eliminate the manual portion of this cleanup procedure entirely and route it through our existing automated cleanup infrastructure, so the class of failure that produced this incident is no longer reachable through human input.\n\n## Closing\n\nReliability and data integrity are foundational to the trust our customers place in Mixpanel, and we recognize the impact this incident had on the teams who rely on us. We are sorry for the disruption. If you have questions about how this incident may have affected a specific project, please reach out to your account team or Mixpanel Support.",
        "createdAt": "2026-05-21T21:43:04.666+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}