Skip to content
OutageDeck

Incident detail

Increased setup failures for T4 models

Increased setup failures for T4 models

Resolved incidentMinor0 affected services

Timeline window

to

Timeline

Incident updates

Updates are normalized from the official source chronology so timeline changes remain easy to scan.

  1. Investigating

    We are seeing increased setup failure rates on models using T4 GPUs.

  2. Investigating

    The problem seems to be isolated to models downloading weights from HuggingFace.

  3. Monitoring

    We are seeing download issues from HuggingFace for a handful of T4 models, stopping new instances from starting. Requests for those models may be delayed as new capacity fails to come online. We will continue to monitor the situation.

  4. Resolved

    Model setup failures have dropped to normal levels since 10:40 UTC. Thank you for your patience!