{"meta":{"version":"v1","pricing":{"public":{"label":"Public","description":"Read-only API access for lightweight status checks and public integrations."},"premium":{"label":"Premium (early access)","description":"API keys with higher hourly quotas are available now; webhooks, alerting, and exports are on the roadmap."}},"generatedAt":"2026-07-09T02:30:28.294Z"},"data":{"id":"incident_gws_KZuQxLyBKvPD1zS6sxVA","slug":"google-workspace-summary-google-chat-users-might-not-be-able-to-open-chat-while-2025-11-24-1zs6sxva","title":"**Summary** Google Chat users might not be able to open chat while connecting from Europe. **Description** We are…","summary":"Summary: Google Chat users might not be able to open chat while connecting from Europe. Description: We are experiencing an issue with Google Chat beginning on Monday, 2025-11-24 00:02 PST. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2025-11-24 03:00 PST with current details. We apologize to all who are affected by the disruption. Symptoms: Google Chat users might not be able to open chat while connecting from Europe. Workaround: None at this time.","status":"resolved","severity":"major","startedAt":"2025-11-24T08:02:00+00:00","updatedAt":"2025-11-27T10:00:50+00:00","resolvedAt":"2025-11-24T09:55:00+00:00","provider":{"slug":"google-workspace","name":"Google Workspace"},"affectedServices":[{"slug":"google-meet","name":"Meet & Chat"}],"links":{"html":"/incidents/google-workspace-summary-google-chat-users-might-not-be-able-to-open-chat-while-2025-11-24-1zs6sxva","api":"/api/v1/incidents/google-workspace-summary-google-chat-users-might-not-be-able-to-open-chat-while-2025-11-24-1zs6sxva","providerHtml":"/providers/google-workspace"},"impactSummary":"Google Workspace reported a medium severity event affecting Google Chat.","source":{"id":"source_google_workspace_status","kind":"official_api","name":"Google Workspace Status Dashboard","checkedAt":"2026-07-09T02:25:02.014+00:00","officialUrl":"https://www.google.com/appsstatus/dashboard","statusPageUrl":"https://www.google.com/appsstatus/dashboard"},"updates":[{"id":"update_gws_KZuQxLyBKvPD1zS6sxVA_20251124t0947180000","status":"investigating","body":"**Summary**\nGoogle Chat users might not be able to open chat while connecting from Europe.\n**Description**\nWe are experiencing an issue with Google Chat beginning on Monday, 2025-11-24 00:02 PST.\nOur engineering team continues to investigate the issue.\nWe will provide an update by Monday, 2025-11-24 03:00 PST with current details.\nWe apologize to all who are affected by the disruption.\n**Symptoms**\nGoogle Chat users might not be able to open chat while connecting from Europe.\n**Workaround**\nNone at this time.","createdAt":"2025-11-24T09:47:18+00:00"},{"id":"update_gws_KZuQxLyBKvPD1zS6sxVA_20251124t1006570000","status":"monitoring","body":"**Summary**\nGoogle Chat users might not be able to open chat while connecting from Europe.\n**Description**\nThe issue with Google Chat has been resolved for all affected users as of Monday, 2025-11-24 01:55 PST.\nWe will publish an analysis of this incident once we have completed our internal investigation.\nWe thank you for your patience while we worked on resolving the issue.\n**Symptoms**\nGoogle Chat users might not be able to open chat while connecting from Europe.\n**Workaround**\nNone at this time.","createdAt":"2025-11-24T10:06:57+00:00"},{"id":"update_gws_KZuQxLyBKvPD1zS6sxVA_20251125t1828130000","status":"resolved","body":"# Preliminary Incident Report\nWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced an impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.\n## Summary\nOn Monday, 24 November 2025, Google Chat experienced an overload which led to service disruptions for users for a duration of 1 hour and 53 minutes. Customers encountered issues including the inability to send messages, load the chat application, and start new conversations. Some notifications were also incorrectly displayed with \"Anonymous\" as the sender\n## Root Cause\nThe trigger of the outage was transient errors served from an internal microservice in Europe that caused Chat backend processes to execute a fallback path to retrieve this data. The root cause was that the data retrieved from the fallback path subsequently caused an update to Chat metadata which were sent to user devices, causing those devices to reload data from Chat backend servers, thereby causing more updates to be sent to user devices, resulting in a cascading effect that overloaded Chat servers in Europe.\nGoogle engineers have implemented a mitigation to limit high rates of requests to prevent this cascading loop from recurring.\n## Remediation\nGoogle engineers were alerted to the incident via alerts and immediately started an investigation.\nTo fully mitigate the impact, our engineering teams took the following steps:\n1. Increased Server Capacity: To handle the unexpected surge in traffic, we significantly increased the capacity of our servers in the affected region to ensure the system could process all incoming requests smoothly.\n2. Limited the Rate of Metadata Updates: To prevent the cascading loop of requests from overloading Chat servers again, Google engineers have limited the rate of metadata updates so that Chat servers will not be overloaded in this way.\n3. Paused Non-Essential Background Tasks: To prioritize the restoration of core chat functions, we temporarily halted several internal background processes. This action helped to reduce the overall load on our systems, freeing up resources to serve user traffic more effectively.\n4. Gradual Traffic Restoration: Our automated defenses had initially blocked some traffic to prevent a complete system overload. A crucial part of the recovery was to gradually and carefully allow this traffic back. This controlled approach ensured that as users reconnected, the system wouldn't be overwhelmed again, allowing everyone to successfully re-establish their connection to Google Chat without getting stuck in a retry loop.\n## Detailed Description of Impact\nFor a duration of 1 hour and 53 minutes, Google Chat users in Europe experienced a service disruption which manifested as an inability to use core features of the service, such as sending messages and starting conversations.","createdAt":"2025-11-25T18:28:13+00:00"},{"id":"update_gws_KZuQxLyBKvPD1zS6sxVA_20251126t2356100000","status":"resolved","body":"# Incident Report\n## Summary\nOn Monday, 24 November 2025, Google Chat experienced an overload which led to service disruptions for users for a duration of 1 hour and 53 minutes. Customers encountered issues including the inability to send messages, load the chat application, and start new conversations. Some notifications were also incorrectly displayed with \"Anonymous\" as the sender.\nWe deeply regret the disruption our Google Workspace customers experienced. We are committed to improving our platform's reliability and performance, and we are taking immediate steps to improve the platform’s performance and availability.\n## Root Cause\nThe trigger of the outage was transient errors served from an internal microservice in Europe that caused Chat backend processes to execute a fallback path to retrieve this data. The root cause was that the data retrieved from the fallback path subsequently caused an update to Chat metadata which were sent to user devices, causing those devices to reload data from Chat backend servers, thereby causing more updates to be sent to user devices resulting in a cascading effect that overloaded Chat servers in Europe.\nGoogle engineers have implemented a mitigation to limit high rates of requests to prevent this cascading loop from recurring.\n## Remediation and Prevention\nGoogle engineers were alerted to the incident via alerts and immediately started an investigation.\nTo fully mitigate the impact, our engineering teams took the following steps:\n1. **Increased Server Capacity:** To handle the unexpected surge in traffic, we significantly increased the capacity of our servers in the affected region to ensure the system could process all incoming requests smoothly.\n2. **Limited the Rate of Metadata Updates**: To prevent the cascading loop of requests from overloading Chat servers again, Google engineers have limited the rate of metadata updates so that Chat servers will not be overloaded in this way.\n3. **Paused Non-Essential Background Tasks:** To prioritize the restoration of core chat functions, we temporarily halted several internal background processes. This action helped to reduce the overall load on our systems, freeing up resources to serve user traffic more effectively.\n4. **Gradual Traffic Restoration:** Our automated defenses had initially blocked some traffic to prevent a complete system overload. A crucial part of the recovery was to gradually and carefully allow this traffic back. This controlled approach ensured that as users reconnected, the system wouldn't be overwhelmed again, allowing everyone to successfully re-establish their connection to Google Chat without getting stuck in a retry loop.\nGoogle is committed preventing a repeat of this issue in the future and is completing the following actions:\n* **Fix the immediate cause**: Stop errors in calls to the microservices from triggering metadata updates to devices. The fix prevents a cascading loop of client devices making requests back to Chat servers resulting in overload.\n* **Prevent devices from sending excessive load**: Implement changes in Chat web and mobile apps to prevent them from performing full reload of data from Chat servers when it is not needed. This reloading contributed greatly to the overload of Chat servers.\n* **Improve Monitoring**: Enhance monitoring to identify which specific metadata updates to user devices cause reloads in order to more quickly mitigate similar issues in the future.\n## Detailed Description of Impact\nOn Monday, 24 November 2025 at 00:02 US/Pacific, Google Chat users in Europe experienced a service disruption affecting core features of the service for a duration of 1 hour and 53 minutes.\nAt its peak, the incident affected approximately 20.2% of users in Europe. The issue was resolved by 01:55 US/Pacific.\nUsers encountered problems with the following core functionality:\n* **Initial Application Load**: Many users had trouble loading the Google Chat application.\n* **Opening Chat Rooms**: Users experienced errors when trying to open existing chat rooms.\n* **Sending Messages**: Users faced delays sending messages.\n* I**ncreased Latency**: There was an increase in message delivery latency for affected users.\n* **Notification Rendering**: Some notifications were also incorrectly displayed with \"Anonymous\" as the sender.","createdAt":"2025-11-26T23:56:10+00:00"}],"access":{"plan":"public","keyed":false}}}