Skip to content

Incident detail

ClickPipes failing on Kinesis in AWS us-east-1

Resolved incidentCritical1 affected service1 region: us-east-1

Timeline window

to

Get alerted the next time ClickHouse Cloud breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Investigating

    Currently investigating failures on AWS us-east-1 for customers using ClickPipes on Kinesis in this region.

  2. Identified

    The issue has been identified as a change on AWS related to Kinesis IAM permissions and appears to be currently isolated to us-east-1. While we work with AWS to mitigate this issue, we recommend to update your IAM policy to include a trailing wildcard (`/*`) to the resource for the `kinesis:RegisterStreamConsumer` action like so:

    ```

    {

    "Effect": "Allow",

    "Action": [

    "kinesis:RegisterStreamConsumer"

    ],

    "Resource": [

    "arn:aws:kinesis:{REGION}:{ACCOUNT_ID}:stream/{STREAM_NAME}/*"

    ]

    }

    ```

    Once this is applied, you must restart your ClickPipe for the changes to take affect.

  3. Identified

    AWS has confirmed the issue on their end and are in the process of rolling back the change. The ETA for this rollback in us-east-1 is by end of day tomorrow. Signs of gradual recovery are expected in us-east-1 as the rollback progresses across AZs during this time. In the meantime, if your ClickPipe is still affected, please update your IAM policy as instructed in the previous status update.

  4. Identified

    AWS has confirmed that on September 15, 2026, beginning at 11:15 AM PDT, customers experienced elevated AccessDeniedException errors when calling the RegisterStreamConsumer API for Amazon Kinesis Data Streams (Kinesis) in the N. Virginia (us-east-1) Region. The root cause was a recent change that introduced a defect in how RegisterStreamConsumer requests are authorized, causing each request to be evaluated against the specific consumer resource (for example, `arn:aws:kinesis:<region>:<account-id>:stream/<stream-name>/consumer/<consumer-name>`) rather than the stream resource (for example, `arn:aws:kinesis:<region>:<account-id>:stream/<stream-name>`). As a result, IAM policies that granted kinesis:RegisterStreamConsumer only at the stream level, which previously succeeded and aligns with our published documentation, no longer matched the request and returned AccessDeniedException.

    AWS has begun rolling back this change in the affected Region, and expect calls that were previously permitted at the stream level to succeed again once the rollback completes on September 16, 2026, by 6:00 PM PDT. As a workaround, please update your IAM policy as instructed in the previous status update.

  5. Identified

    AWS has confirmed that on September 15, 2026, beginning at 11:15 AM PDT, customers experienced elevated AccessDeniedException errors when calling the RegisterStreamConsumer API for Amazon Kinesis Data Streams (Kinesis) in the N. Virginia (us-east-1) Region. The root cause was a recent change that introduced a defect in how RegisterStreamConsumer requests are authorized, causing each request to be evaluated against the specific consumer resource (for example, `arn:aws:kinesis:<region>:<account-id>:stream/<stream-name>/consumer/<consumer-name>`) rather than the stream resource (for example, `arn:aws:kinesis:<region>:<account-id>:stream/<stream-name>`). As a result, IAM policies that granted kinesis:RegisterStreamConsumer only at the stream level, which previously succeeded and aligns with our published documentation, no longer matched the request and returned AccessDeniedException.

    AWS has begun rolling back this change in the affected Region, and expect calls that were previously permitted at the stream level to succeed again once the rollback completes on September 16, 2026, by 6:00 PM PDT. As a workaround, please update your IAM policy as instructed in the previous status update.

  6. Resolved

    This incident has been resolved. AWS has confirmed that the Enhanced Fanhout (EFO) Kinesis pipes in us-east-1 region are fully operational. Please contact ClickHouse Support if you experience any further issues: <https://console.clickhouse.cloud/support>

Keep exploring

More from ClickHouse Cloud

Neighboring incidents on ClickHouse Cloud's timeline and the rest of their record on OutageDeck.