{
  "meta": {
    "version": "v1",
    "pricing": {
      "public": {
        "label": "Public",
        "description": "Read-only API access for lightweight status checks and public integrations."
      },
      "premium": {
        "label": "Premium",
        "description": "API keys with higher hourly quotas, plus Slack, Teams, Discord, webhook, and email outage alerts across your vendor stack."
      }
    },
    "generatedAt": "2026-11-17T23:02:59.576Z"
  },
  "data": {
    "id": "incident_statuspage_frontegg_y4rt16yjv7fd",
    "slug": "frontegg-eu-degraded-state-partial-outage-2024-07-24",
    "title": "EU Degraded State - Partial Outage",
    "summary": "EU Degraded State - Partial Outage",
    "status": "resolved",
    "stale": false,
    "severity": "major",
    "startedAt": "2024-07-24T08:51:54.44+00:00",
    "updatedAt": "2024-07-26T14:00:53.15+00:00",
    "resolvedAt": "2024-07-24T10:58:00.432+00:00",
    "provider": {
      "slug": "frontegg",
      "name": "Frontegg"
    },
    "affectedServices": [],
    "links": {
      "html": "/incidents/frontegg-eu-degraded-state-partial-outage-2024-07-24",
      "api": "/api/v1/incidents/frontegg-eu-degraded-state-partial-outage-2024-07-24",
      "providerHtml": "/providers/frontegg",
      "alerts": "https://outagedeck.com/account?stack=frontegg&utm_source=api&utm_medium=response&utm_campaign=api_alerts&utm_content=incident"
    },
    "impactSummary": "Frontegg reported a major event for the affected tracked services.",
    "source": {
      "id": "source_frontegg_status",
      "kind": "official_api",
      "name": "Frontegg Status",
      "checkedAt": "2026-11-17T23:00:12.229+00:00",
      "stale": false,
      "officialUrl": "https://status.frontegg.com",
      "statusPageUrl": "https://status.frontegg.com"
    },
    "updates": [
      {
        "id": "update_statuspage_frontegg_y4rt16yjv7fd_131kgdcz01w2",
        "status": "investigating",
        "body": "We are currently investigating this issue.",
        "createdAt": "2024-07-24T08:51:54.503+00:00"
      },
      {
        "id": "update_statuspage_frontegg_y4rt16yjv7fd_yl9tm23f0m75",
        "status": "identified",
        "body": "The issue has been identified and a fix is being implemented.",
        "createdAt": "2024-07-24T09:06:07.036+00:00"
      },
      {
        "id": "update_statuspage_frontegg_y4rt16yjv7fd_x2f59l4nxltj",
        "status": "monitoring",
        "body": "A fix has been implemented and we are monitoring the results.",
        "createdAt": "2024-07-24T09:13:43.657+00:00"
      },
      {
        "id": "update_statuspage_frontegg_y4rt16yjv7fd_2rmy5wx9g4mp",
        "status": "monitoring",
        "body": "We are continuing to monitor for any further issues.",
        "createdAt": "2024-07-24T09:47:13.203+00:00"
      },
      {
        "id": "update_statuspage_frontegg_y4rt16yjv7fd_fwq3tvx8f3kw",
        "status": "monitoring",
        "body": "We are continuing to monitor for any further issues.",
        "createdAt": "2024-07-24T10:37:27.425+00:00"
      },
      {
        "id": "update_statuspage_frontegg_y4rt16yjv7fd_fyjkg8rlxry4",
        "status": "resolved",
        "body": "This incident has been resolved.",
        "createdAt": "2024-07-24T10:58:00.432+00:00"
      },
      {
        "id": "update_statuspage_frontegg_y4rt16yjv7fd_n8hkpfqny98p",
        "status": "resolved",
        "body": "# **Root Cause Analysis \\(RCA\\) Report**\n\n**Date and Time**: July 24, 2024**Duration**: 22 minutes **Affected Services**: Authentication and core services**Impact**: Customers in the EU region were hanging and returned as 504 timeouts**Reported By**: Internal monitoring systems and customers\n\n\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\\_\n\n**Executive summary:**\n\nOn Wednesday, July 24th, at 08:43 GMT, Frontegg's internal monitoring systems indicated that the API Gateway encountered an issue following the deployment of a new OpenTelemetry propagator \\(OTEL instrumentation\\), causing service disruptions in the EU. As a result, some of our customers were experiencing timeout errors \\(HTTP status 504\\) returned by Frontegg.\n\nDuring the upgrade of our API Gateway, Frontegg also updated the OpenTelemetry library. This update inadvertently caused the system to send data one piece at a time instead of using efficient batches due to a misconfiguration in the data handling settings.\n\nOTEL transmitted millions of traces individually rather than in aggregated batches. Although our system was rigorously tested under various conditions, the high load in the EU environment caused our auto-scaling mechanism to lag behind the incoming traffic. This led to the API gateway being overwhelmed by the volume of client requests.\n\n‌\n\n**Cause Analysis:**\n\nThe primary cause of the incident was the deployment of a new OTEL instrumentation in the API Gateway, which led to a significant increase in trace data volume. Contributing factors included:\n\n* The API Gateway's OTEL was configured with the BasicPropagator instead of a BatchPropogator, sending each trace as part of the flow.\n* The fast rise of HTTP requests to the OTEL collector overloaded the API gateway to handle incoming requests. Although it was autoscaled, it lacked in response to the number of requests.\n* With the increase of traces being sent, the OTEL Collector failed to handle millions of traces at such a rate, increasing the request handling time, which caused another increase in API-gateway HTTP requests\n\n‌\n\n**Customer Impact**\n\nDuring the incident, customers in the European region experienced significant service degradation. Specific issues included failures in hosted login monitors and general service instability.\n\n‌\n\n**Mitigation and resolution:**\n\nUpon receiving the initial alerts, the Frontegg team began investigating the issue promptly. After identifying the problem with the OTEL propagator and collector, we increased the allocated resources and reverted to the latest working version. Following the implementation of this change, the systems returned to normal operations.\n\n**Mitigation**:\n\n* Increased the CPU allocation for the OTEL Gateway to handle the increased workload.\n* Revert to the latest Api-gateway version.\n\n**Resolution**:\n\n* Restarted the API Gateway to clear hanging requests and stabilize the OTEL Gateway.\n* Deployed a new version of the API gateway with the correct configuration\n\n‌\n\n**Prevention and Future steps:** Enhance OTEL Propagator: Implement batch processing, asynchronous handling, and strict timeouts.\n\n* **Upgrade OTEL Gateway**: Allocate additional resources to the OTEL Gateway and implement autoscaling to handle increased workloads effectively.\n* **Implement Aggressive Timeouts**: Implement stringent timeout policies for all HTTP requests that are not customer-related. This measure will proactively prevent delays and mitigate the risk of unresponsive requests.\n* Stress tests: change the deployment pipeline to include stress testing instead of the nightly testing suite.\n\n‌\n\n**Communication:**\n\n**Enhance Status Page Communication**: Ensure the status page provides clear and timely updates during incidents. Develop and maintain standardized templates for incident communication to facilitate prompt and consistent information, even if the root cause is not immediately identified.",
        "createdAt": "2024-07-26T14:00:47.574+00:00"
      }
    ],
    "access": {
      "plan": "public",
      "keyed": false
    }
  }
}