{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Teams, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-11-18T19:13:31.989Z"
  },
  "data": {
    "id": "incident_statuspage_upstash_x1cpxkl3lby2",
    "slug": "upstash-performance-degradation-on-qstash-2024-11-13",
    "title": "Performance degradation on QStash",
    "summary": "Performance degradation on QStash",
    "status": "resolved",
    "stale": false,
    "severity": "major",
    "startedAt": "2024-11-13T14:14:40.844+00:00",
    "updatedAt": "2024-11-13T20:48:41.77+00:00",
    "resolvedAt": "2024-11-13T17:27:29.698+00:00",
    "provider": {
      "slug": "upstash",
      "name": "Upstash"
    },
    "affectedServices": [
      {
        "slug": "upstash-qstash-vector",
        "name": "QStash & Vector"
      }
    ],
    "regions": [
      "eu-central-1"
    ],
    "links": {
      "html": "/incidents/upstash-performance-degradation-on-qstash-2024-11-13",
      "api": "/api/v1/incidents/upstash-performance-degradation-on-qstash-2024-11-13",
      "providerHtml": "/providers/upstash",
      "alerts": "https://outagedeck.com/account?stack=upstash&utm_source=api&utm_medium=response&utm_campaign=api_alerts&utm_content=incident"
    },
    "impactSummary": "Upstash reported a major event for the affected tracked services.",
    "source": {
      "id": "source_upstash_status",
      "kind": "official_api",
      "name": "Upstash Status",
      "checkedAt": "2026-11-18T19:10:10.245+00:00",
      "stale": false,
      "officialUrl": "https://status.upstash.com",
      "statusPageUrl": "https://status.upstash.com"
    },
    "updates": [
      {
        "id": "update_statuspage_upstash_x1cpxkl3lby2_yc5s8zlx9wqc",
        "status": "investigating",
        "body": "We are currently investigating this issue.",
        "createdAt": "2024-11-13T14:14:40.901+00:00"
      },
      {
        "id": "update_statuspage_upstash_x1cpxkl3lby2_468dhz170sjv",
        "status": "identified",
        "body": "The issue has been identified and a fix is being implemented.",
        "createdAt": "2024-11-13T15:38:44.419+00:00"
      },
      {
        "id": "update_statuspage_upstash_x1cpxkl3lby2_w4281hkvftgs",
        "status": "resolved",
        "body": "We will be sharing a postmortem about the incident soon.",
        "createdAt": "2024-11-13T17:27:29.698+00:00"
      },
      {
        "id": "update_statuspage_upstash_x1cpxkl3lby2_pv7nng9srkg1",
        "status": "resolved",
        "body": "**Product:** QStash  \n**Impact:** Degraded performance, delayed processing of events, and duplicate event deliveries for some customers\n\n## Incident Summary\n\nQStash experienced an incident marked by a sudden and extreme load on our servers. This caused a degradation in performance, with extremely high latency for event processing for all users. We also noticed some of the events being delivered multiple times to some of the users. To mitigate the high load, we have increased the capacity as our initial response while investigation proceeds. Eventually, fixes for the issues are confirmed with an issue reproducer and deployed to production.\n\n## Root Cause Analysis\n\nIn a certain type of usage, failure handling of [failureFunction](https://upstash.com/docs/workflow/basics/serve#failurefunction) can cause recursive calls which causes a leak in the queue of the tasks, causing a severe load on the QStash servers. This also triggered an edge case which caused some of the events to be delivered multiple times.\n\n## Resolution\n\nTwo hotfixes to the QStash processes are deployed\n\n1. Prevent recursive calls within the failure function.  \n2. Eliminate duplicate deliveries while keeping \"at least once delivery\" guarantee.  \n  These are verified to successfully resolve the root cause, normalizing server load and restoring standard event processing operations.\n\n## Impact on Customers\n\nHigh latency of event processing is observed for all users. Some users received duplicate event deliveries. No events were lost, and all were delivered as part of our \"at least once delivery\" guarantee. Customers do not need to take any corrective action, as workflows have returned to normal and preventive fixes are deployed.",
        "createdAt": "2024-11-13T20:38:27.548+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}