Skip to content

Incident detail

Query failures in Dremio Cloud (US & EU)

Resolved incidentMajor2 affected services

Timeline window

to

Get alerted the next time Dremio breaks

Free email alerts for the handful of vendors you cannot afford to miss. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update on the official status source, oldest to newest, exactly as it appeared there.

  1. Investigating

    We are investigating issues with queries in Dremio Cloud (US & EU). We will provide an update within the next 30 minutes.

  2. Investigating

    We are continuing to investigate issues with queries in Dremio Cloud (US & EU). We have engaged multiple engineering teams to investigate the issue and will provide an update within the next 30 minutes.

  3. Investigating

    We are continuing to investigate issues with queries in Dremio Cloud (US & EU). We have engaged multiple engineering teams to investigate and narrow down the issue. We will provide an update within the next 30 minutes.

  4. Investigating

    We are continuing to investigate the issue with queries in Dremio Cloud (US & EU). Engineering has narrowed down the issue and are working on multiple possible mitigations. We will provide an update within the next 45 minutes.

  5. Investigating

    We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

  6. Investigating

    We have narrowed down the issue impacting queries in Dremio Cloud (US & EU). Engineering is in the process of building and testing a fix to address the issue. We will provide an update within the next 45 minutes.

  7. Identified

    We have identified the issue impacting queries in Dremio Cloud (US & EU) and are in the process of deploying a fix to address the issue. We will provide an update within the next 45 minutes.

  8. Resolved

    Incident Summary

    We experienced a service disruption starting at 1:15 PM PST that resulted in query failures when using reflections in multiple regions. We sincerely apologize for the impact this had on your workflows and the inconvenience it may have caused.

    Incident Details

    An update to the permissions used by Dremio internal sources inadvertently restricted access to some of the folders in the project store, resulting in queries using reflections or accessing those folders to fail in some AWS projects under certain conditions. Our team began investigating the issue immediately and was able to identify the issue and implement a fix at 5 PM PST. Concurrently, we encountered cloud provider issues in our Azure deployment infrastructure which impacted our ability to roll out the fix to our production environments. Once we were able to resolve issue, we were able to validate and deploy a fix to fully restore the service at 10:35 PM PST.

    Our engineers are monitoring the system to ensure full stability. In addition, we are also implementing new alerts to detect such situation quickly in the future.

Keep exploring

More from Dremio

Neighboring incidents on Dremio's timeline and the rest of their record on OutageDeck.