Provider
DatadogIncident detail
Notifications to Pagerduty, VictorOps, Slack, Webhook, Microsoft Teams and OpsGenie are not delivered
Resolved incidentMajor1 affected service
Timeline window
to
Outage alerts
Get alerted the next time Datadog breaks
Free email alerts for up to 5 providers. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.
Timeline
Incident updates
Every update Datadog posted, oldest to newest, exactly as it appeared on their official status page.
Investigating
We are investigating an issue to send notifications to Pagerduty, VictorOps, Slack, Webhook, Microsoft Teams and OpsGenie.
Identified
The issue has been identified and a fix is being implemented.
Identified
We are continuing to work on a fix for this issue.
Identified
New notifications are now successfully delivered, and we are catching-up on past notifications that may have been delayed.
Monitoring
New notifications are now successfully delivered, and we are catching-up on past notifications that may have been delayed
Monitoring
We are still catching-up on past notifications that may have been delayed.
Monitoring
We are still catching-up on past notifications that may have been delayed.
Monitoring
Notifications for Opsgenie, VictorOps, BigPanda, Zendesk, and Sumologic are caught up. Webhooks, Microsoft Teams, Slack, JIRA, and ServiceNow are still catching up.
Monitoring
Notifications for JIRA are now caught up.
Monitoring
Notifications for ServiceNow and Microsoft Teams are now caught up. Webhooks and Slack notifications are still catching up.
Monitoring
Webhooks and Slack notifications are still catching up.
Monitoring
Webhooks and Slack notifications are still catching up.
Monitoring
Webhooks and Slack notifications are still catching up.
Resolved
Between approximately 9:42 AM and 10:38 AM ET on April 10, we observed delivery failures for customers in the US1, AP1, and AP2 regions. During this window, notifications to the following integrations were delayed or temporarily undelivered:
• PagerDuty
• Slack
• Microsoft Teams
• Webhooks
• Jira
• ServiceNow
• OpsGenie
• VictorOps
• BigPanda
• Zendesk
• Sumologic
All delayed notifications have been replayed with the following exceptions: PagerDuty pages and Slack notifications queued during the outage window were not replayed, as they were no longer actionable at the time of recovery. Webhook notifications will be reprocessed and delivered at a later date.
No monitoring data was lost during this incident. Monitor evaluations continued normally — only the delivery of triggered notifications was affected.
Thank you for your patience.
Keep exploring
More from Datadog
Neighboring incidents on Datadog's timeline and the rest of their record on OutageDeck.