Skip to content
OutageDeck

Incident detail

Creation of new instances in AWS is delayed or stuck

Resolved incidentMinor

Timeline window

to

Outage alerts

Get alerted the next time ClickHouse Cloud breaks

Free email alerts for up to 5 providers — no card, live in about a minute. Paid plans add Slack, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Set up free alertsHow ClickHouse Cloud alerts work

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Investigating

    We spotted an issue where creation of new ClickHouse instances is delayed or stuck. Team is investigating the issue and already applying a temporary remediation.

  2. Identified

    We identified the issue and applying the remediation. The issue was caused by manual operation performed on the AWS Route53 configuration earlier today. It caused increase in the rate of requests to Route53 API, leading to throttling and retries (including provisioning of new ClickHouse instances).

  3. Monitoring

    We applied the fix for a subset of regions and see signs of improvements. We are gradually applying the fix to all other regions and monitoring the status.

  4. Monitoring

    ClickHouse continues to scale external-dns in each region. The following regions are complete and operating normally: us-east-1, af-south-1, ap-east-1, ap-northeast-1, ap-northeast-2, ap-south-1

    Additional regions are in progress or will begin shortly, and the ETA to complete the remaining regions is estimated as 3-5 hours: us-east-2, us-west-2, eu-central-1, eu-west-1, eu-west-2, ap-southeast-1, ap-southeast-2, il-central-1

  5. Resolved

    This issue is resolved. External-DNS has been scaled for all impacted regions.