{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Teams, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-10-12T20:16:23.585Z"
  },
  "data": {
    "id": "incident_statuspage_upstash_9vxb0srq33b0",
    "slug": "upstash-qstash-us-region-schedule-degradation-2026-05-05",
    "title": "QStash US Region: Schedule Degradation",
    "summary": "QStash US Region: Schedule Degradation",
    "status": "resolved",
    "stale": false,
    "severity": "minor",
    "startedAt": "2026-05-05T14:24:43.018+00:00",
    "updatedAt": "2026-05-06T08:56:42.638+00:00",
    "resolvedAt": "2026-05-06T08:05:27.074+00:00",
    "provider": {
      "slug": "upstash",
      "name": "Upstash"
    },
    "affectedServices": [
      {
        "slug": "upstash-qstash-vector",
        "name": "QStash & Vector"
      }
    ],
    "links": {
      "html": "/incidents/upstash-qstash-us-region-schedule-degradation-2026-05-05",
      "api": "/api/v1/incidents/upstash-qstash-us-region-schedule-degradation-2026-05-05",
      "providerHtml": "/providers/upstash",
      "alerts": "https://outagedeck.com/account?stack=upstash&utm_source=api&utm_medium=response&utm_campaign=api_alerts&utm_content=incident"
    },
    "impactSummary": "Upstash reported a minor event for the affected tracked services.",
    "source": {
      "id": "source_upstash_status",
      "kind": "official_api",
      "name": "Upstash Status",
      "checkedAt": "2026-10-12T20:10:10.062+00:00",
      "stale": false,
      "officialUrl": "https://status.upstash.com",
      "statusPageUrl": "https://status.upstash.com"
    },
    "updates": [
      {
        "id": "update_statuspage_upstash_9vxb0srq33b0_skbpk13qcryg",
        "status": "identified",
        "body": "We are currently experiencing issues in the US region.\n\n- Duplicate Deliveries: During this period, some scheduled jobs may be executed twice.\n- Schedule Disruption: Schedules created between April 24, 2026 and May 2, 2026 are currently not running.\n\nOur team is actively working on a fix. Once the migration is complete, affected schedules will resume normal operation.\n\nWe will provide updates as progress continues.",
        "createdAt": "2026-05-05T14:24:43.067+00:00"
      },
      {
        "id": "update_statuspage_upstash_9vxb0srq33b0_kfcztx2gsbjv",
        "status": "identified",
        "body": "We are continuing to work on a fix for this issue.",
        "createdAt": "2026-05-05T14:29:32.789+00:00"
      },
      {
        "id": "update_statuspage_upstash_9vxb0srq33b0_5k4f3js2vg83",
        "status": "monitoring",
        "body": "A fix has been implemented and we are monitoring the results.",
        "createdAt": "2026-05-05T15:44:33.635+00:00"
      },
      {
        "id": "update_statuspage_upstash_9vxb0srq33b0_26ktv4z6vrnl",
        "status": "monitoring",
        "body": "Main schedule functionality is back to normal. We are currently checking if previously created schedules are delivered as expected before marking the incident as resolved.",
        "createdAt": "2026-05-05T21:31:22.484+00:00"
      },
      {
        "id": "update_statuspage_upstash_9vxb0srq33b0_lcwv81wrk04n",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2026-05-06T08:05:27.074+00:00"
      },
      {
        "id": "update_statuspage_upstash_9vxb0srq33b0_gq2frtwj0l5v",
        "status": "resolved",
        "body": "# **Incident Postmortem: Scheduled Jobs Inconsistency in US Region**\n\nOn May 1, 2026, we experienced an incident affecting a subset of schedules in the US region following a recent infrastructure update.\n\nThe issue has been resolved, and all affected schedules have been restored.\n\n## **Summary**\n\nAs part of an ongoing scalability improvement, we recently updated scheduling infrastructure in the US region to a new architecture. During this transition, a legacy execution path remained in the codebase as a fallback mechanism.\n\nOn May 1, a bug caused the system to revert to the legacy path. This resulted in inconsistent state between the old and new scheduling systems for some users.\n\n## **Impact**\n\nThe incident affected a limited number of users in the US region.\n\n**Most users were not affected**, and the vast majority of schedules continued operating normally throughout the incident.\n\nUsers who did not update schedules during the transition window continued operating normally throughout the incident.\n\n**A subset of users who created, edited, paused, or deleted schedules between April 24 and May 1 may have experienced one or more of the following:**\n\n* Schedule updates not being reflected\n* Paused schedules becoming active again\n* Deleted schedules reappearing\n* Newly created schedules not executing as expected\n\nSchedules created after the transition may have stopped executing briefly before recovery.\n\n## **Root Cause**\n\nDuring the transition, the new scheduling infrastructure became the source of truth for schedule state.\n\nDue to a bug, the system unexpectedly reverted traffic to the legacy scheduling path, which began accepting updates independently from the new system.\n\nThis caused the two systems to diverge and resulted in inconsistent schedule state for affected users.\n\n## **Resolution**\n\nAfter identifying the issue, we:\n\n1. Restored the new scheduling system as the active source of truth\n2. Reconciled data between the legacy and new systems\n3. Updated missing schedule changes back into the new infrastructure\n4. Performed conflict resolution to preserve user data and schedule continuity\n\nIn some cases, schedules that had previously been paused or deleted were restored to avoid permanent data loss.\n\n## **Preventive Measures**\n\nWe are implementing several changes to prevent similar incidents:\n\n* Removing obsolete fallback execution paths after transitions complete\n* Adding automated safeguards and alerts for unexpected system fallback behavior\n* Improving consistency validation between systems\n* Expanding rollback and reconciliation testing\n\nWe apologize for the disruption and appreciate everyone’s patience while we resolved the issue.",
        "createdAt": "2026-05-06T08:55:34.694+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}