{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Teams, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-09-06T12:37:06.696Z"
  },
  "data": {
    "id": "incident_statuspage_harness_v8lfzxz8cjvr",
    "slug": "harness-aidi-dashboards-degraded-performance-2026-07-22",
    "title": "AIDI Dashboards – Degraded Performance",
    "summary": "AIDI Dashboards – Degraded Performance",
    "status": "resolved",
    "severity": "minor",
    "startedAt": "2026-07-22T14:55:15.264+00:00",
    "updatedAt": "2026-08-10T18:24:01.842+00:00",
    "resolvedAt": "2026-07-22T23:16:51.506+00:00",
    "provider": {
      "slug": "harness",
      "name": "Harness"
    },
    "affectedServices": [],
    "links": {
      "html": "/incidents/harness-aidi-dashboards-degraded-performance-2026-07-22",
      "api": "/api/v1/incidents/harness-aidi-dashboards-degraded-performance-2026-07-22",
      "providerHtml": "/providers/harness",
      "alerts": "https://outagedeck.com/account?stack=harness&utm_source=api&utm_medium=response&utm_campaign=api_alerts&utm_content=incident"
    },
    "impactSummary": "Harness reported a minor event for the affected tracked services.",
    "source": {
      "id": "source_harness_status",
      "kind": "official_api",
      "name": "Harness Status",
      "checkedAt": "2026-09-06T12:35:06.768+00:00",
      "officialUrl": "https://status.harness.io",
      "statusPageUrl": "https://status.harness.io"
    },
    "updates": [
      {
        "id": "update_statuspage_harness_v8lfzxz8cjvr_qs5lgrqknqq6",
        "status": "investigating",
        "body": "We are investigating an issue impacting AIDI dashboards. Users may experience increased load times or intermittent failures when accessing dashboards. Our team is actively working to identify the root cause and restore normal performance. We will provide updates as more information becomes available.",
        "createdAt": "2026-07-22T14:55:15.445+00:00"
      },
      {
        "id": "update_statuspage_harness_v8lfzxz8cjvr_cxbdlf237c5w",
        "status": "identified",
        "body": "The issue has been identified and a fix is being implemented.",
        "createdAt": "2026-07-22T14:58:00.233+00:00"
      },
      {
        "id": "update_statuspage_harness_v8lfzxz8cjvr_tl1rqwsnnhgf",
        "status": "monitoring",
        "body": "A fix has been implemented and we are monitoring the results.",
        "createdAt": "2026-07-22T15:25:34.153+00:00"
      },
      {
        "id": "update_statuspage_harness_v8lfzxz8cjvr_y7hsvd83vj1v",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2026-07-22T23:16:51.506+00:00"
      },
      {
        "id": "update_statuspage_harness_v8lfzxz8cjvr_qs5kgbq90ky3",
        "status": "resolved",
        "body": "## Summary\n\nCustomers on Prod1, Prod2, and Prod3 clusters experienced intermittent widget load failures and increased load times when accessing AIDI 2.0 dashboards on July 22, 2026. Not all widgets were affected simultaneously  the issue manifested as sporadic failures rather than a full outage.\n\nNo customer data was lost. SEI 1.0 customers were not impacted.\n\n## Root Cause\n\nOver time, a routine database maintenance process failed to run on certain tables in our analytics database, causing those tables to accumulate a large volume of internal metadata used to track deleted records. When the database planned queries against these tables, it loaded all of this accumulated metadata into memory, causing memory usage on the affected nodes to spike repeatedly. These repeated spikes triggered an automatic safety mechanism that restarts a node when it detects excessive memory pressure, and the affected nodes began restarting in a loop as a result. This caused intermittent, degraded query performance on AIDI 2.0 dashboards for the duration of the incident.\n\n## Impact\n\nCustomers on Prod1, Prod2, and Prod3 clusters may have experienced intermittent widget load failures or increased load times on AIDI 2.0 dashboards.\n\n**Duration:** July 22, 2026, 07:58 PDT – 16:16 PDT \\(~8 hours 18 minutes\\), with intermittent widget failures; system was restarted and under active monitoring from 08:25 PDT onward.\n\n### What was not impacted?\n\n* Data ingestion and processing\n* SEI 1.0 customers\n* Integrations and metadata flows\n\nNo customer data was lost.\n\n## Remediation\n\nUpon identifying the issue, the affected database nodes were restarted at 08:25 PDT, which restored initial stability. We continued to monitor the system closely, and when intermittent degradation was still observed afterward, we applied several additional fixes:\n\n* Adjusted database configuration settings to limit the amount of memory used for processing accumulated metadata, and tuned query-planning settings to reduce memory pressure.\n* Ran cleanup jobs to reduce the backlog of accumulated metadata on the affected tables.\n* Increased capacity on the affected database nodes to provide additional headroom.\n\nThese changes progressively stabilized the system, and the incident was fully resolved at 16:16 PDT.\n\n## Action Items\n\nTo prevent recurrence, we are implementing the following:\n\n1. We have upgraded the backend which includes underlying improvements that handle memory spikes caused by excessive delete files.\n2. We have rolled out automated  compaction jobs for newly introduced tables to prevent delete file accumulation going forward.",
        "createdAt": "2026-08-10T18:17:49.978+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}