{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Teams, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-11-17T23:05:39.057Z"
  },
  "data": {
    "id": "incident_statuspage_upstash_kg3gf7hftk83",
    "slug": "upstash-qstash-degraded-performance-in-request-processing-and-event-logs-2025-04-02",
    "title": "QStash: Degraded performance in request processing and event logs",
    "summary": "QStash: Degraded performance in request processing and event logs",
    "status": "resolved",
    "stale": false,
    "severity": "minor",
    "startedAt": "2025-04-02T07:15:34+00:00",
    "updatedAt": "2025-04-03T15:58:24.366+00:00",
    "resolvedAt": "2025-04-02T21:12:51.042+00:00",
    "provider": {
      "slug": "upstash",
      "name": "Upstash"
    },
    "affectedServices": [
      {
        "slug": "upstash-qstash-vector",
        "name": "QStash & Vector"
      }
    ],
    "regions": [
      "eu-central-1"
    ],
    "links": {
      "html": "/incidents/upstash-qstash-degraded-performance-in-request-processing-and-event-logs-2025-04-02",
      "api": "/api/v1/incidents/upstash-qstash-degraded-performance-in-request-processing-and-event-logs-2025-04-02",
      "providerHtml": "/providers/upstash",
      "alerts": "https://outagedeck.com/account?stack=upstash&utm_source=api&utm_medium=response&utm_campaign=api_alerts&utm_content=incident"
    },
    "impactSummary": "Upstash reported a minor event for the affected tracked services.",
    "source": {
      "id": "source_upstash_status",
      "kind": "official_api",
      "name": "Upstash Status",
      "checkedAt": "2026-11-17T23:00:10.968+00:00",
      "stale": false,
      "officialUrl": "https://status.upstash.com",
      "statusPageUrl": "https://status.upstash.com"
    },
    "updates": [
      {
        "id": "update_statuspage_upstash_kg3gf7hftk83_1qq4kj1qm3x9",
        "status": "investigating",
        "body": "We are currently investigating this issue.",
        "createdAt": "2025-04-02T07:35:34.246+00:00"
      },
      {
        "id": "update_statuspage_upstash_kg3gf7hftk83_9bf4lys5bq1f",
        "status": "monitoring",
        "body": "A fix has been implemented and we are monitoring the results.",
        "createdAt": "2025-04-02T08:20:18.955+00:00"
      },
      {
        "id": "update_statuspage_upstash_kg3gf7hftk83_ky4ppbfzrct7",
        "status": "investigating",
        "body": "We are currently investigating this issue.",
        "createdAt": "2025-04-02T16:21:16.112+00:00"
      },
      {
        "id": "update_statuspage_upstash_kg3gf7hftk83_dbynm0mvxwr2",
        "status": "investigating",
        "body": "We are continuing to investigate this issue.",
        "createdAt": "2025-04-02T16:29:46.228+00:00"
      },
      {
        "id": "update_statuspage_upstash_kg3gf7hftk83_0lgb93ml52y1",
        "status": "investigating",
        "body": "We are continuing to investigate this issue.",
        "createdAt": "2025-04-02T16:33:35.186+00:00"
      },
      {
        "id": "update_statuspage_upstash_kg3gf7hftk83_87k8rclfk1sb",
        "status": "monitoring",
        "body": "Main QStash service is back to normal. \nEvent logs are still temporarily unavailable.",
        "createdAt": "2025-04-02T17:33:09.167+00:00"
      },
      {
        "id": "update_statuspage_upstash_kg3gf7hftk83_1dpxv1hd6ny9",
        "status": "monitoring",
        "body": "Monitoring: \nMain QStash service is back to normal. \nEvent logs service is back online but events will be lagging a few mins.",
        "createdAt": "2025-04-02T20:20:43.828+00:00"
      },
      {
        "id": "update_statuspage_upstash_kg3gf7hftk83_cds81489ssz2",
        "status": "resolved",
        "body": "QStash service and Event logs are fully functional without any remaning issues.",
        "createdAt": "2025-04-02T21:12:51.042+00:00"
      },
      {
        "id": "update_statuspage_upstash_kg3gf7hftk83_ls8nt3q9pvfz",
        "status": "resolved",
        "body": "###   \nProduct: QStash\n\n### Incident Summary\n\nDue to high load, the volume of QStash event logs reached to a point which caused latency in the underlying data store operations.  \nEvent log creation was slowed down and lead to performance degradation in QStash request processing.  \nIn order to resolve the performance degradation in QStash requests, event logging module was turned off temporarily.  \nAfter deploying a hot fix and configuration changes, we eventually turned on event logging and system went back to stable state again.\n\n### Root Cause\n\nAt 07:15 UTC we received alerts on the performance degradation and started the investigation.  \nWe discovered long running queries for synching event logs from main QStash servers to QStash event server.  \nIn order to resolve performance degradation, we turned off event logging functionality as an immediate action.  \nThis action turned the performance back to normal levels for QStash requests but left event log processing disabled.  \nWe deployed a hotfix during the day to remove some redundant calls and alleviate the impact.  \nAround 16:20 UTC, we observed another performance degradation on QStash requests due to a load increase, and disabled event log processing again.  \nIn the following hours, we deployed a configuration change to relax the job interval durations for event log tasks and turned on event logging again.  \nThis configuration change helped to resolve the performance issues without any further issues.\n\n### Impact\n\nDuring the problematic timeframes, when the slow event log processing was observed, QStash requests experienced high latency and caused timeouts for customers.  \nNo events were lost. Duplicate event deliveries were observed due to a number of restarts during the incident.\n\n### Resolution\n\nImprovements are applied to the event logging module to prevent the same issue from happening again.  \nAlso, we have planned to upgrade underlying disks to stronger models.",
        "createdAt": "2025-04-03T15:49:18.956+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}