Skip to content

Incident detail

Delays in notification service

Resolved incidentCritical1 affected service

Timeline window

to

Get alerted the next time Opsgenie breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Investigating

    We are seeing delays with outbound notifications. We have identified the cause and are currently working on mitigation of this issue.

  2. Identified

    We have identified the problem and working on it. We are expecting that notification service will return normal state in a short time.

  3. Monitoring

    A fix has been implemented and we are monitoring the results.

  4. Resolved

    This incident has been resolved.

  5. Resolved

    ### SUMMARY

    On Sep 14, 2022, between 03:36 PM and 04:26 PM UTC, Atlassian customers using the Opsgenie product received delayed notifications for up to 50 minutes. The event was triggered by a code change that upgrades a common framework. The changes included in this framework update impacted customers in the both US and EU regions. The incident was detected  by the on-call developer and mitigated by reverting the latest changes, which put Opsgenie systems into a known good state. The total time to resolution was around 50 minutes.

    ### IMPACT

    The overall impact was between Sep 14, 2022, 03:36 PM UTC, and Sep 14, 2022, 04:26 PM UTC on Opsgenie products. The incident service disruption was limited to US and EU region customers who did not receive their notifications immediately, but instead experienced notification delays of up to 50 minutes. In total, ~132K notifications in the US region and ~23.6K notifications in the EU region were sent with delays. Only less than %0.6 of the active customers were affected.

    ### ROOT CAUSE

    The issue was caused by an Atlassian-initiated change to upgrade a common framework. While the majority of the intended changes had been tested successfully, there were some accompanying changes with the framework upgrade that caused the notification service to stop processing new notification requests. Instead, these notifications remained in the queues until the deployment was reverted, resulting in notification delays for customers of up to 50 minutes.

    ### REMEDIAL ACTIONS PLAN & NEXT STEPS

    We know that outages impact your productivity.

    We are prioritizing the following improvement actions to avoid repeating this type of incident:

    • We are improving the testing and deployment processes we follow after framework updates.
    • We are implementing new monitoring to reduce the detection and response time even further.

    We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.

    Thanks,

    Atlassian Customer Support

Keep exploring

More from Opsgenie

Neighboring incidents on Opsgenie's timeline and the rest of their record on OutageDeck.