We are fully down for a subset of customers. We're investigating the issue, and are rolling back a deployment.

We are currently investigating this issue.

We are continuing to investigate this issue.

The rollback seems to have completely resolved errors. We're monitoring closely, but users should be able to access Asana again.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Incident: An internal system responsible for automatically scaling backend server capacity in one of our compute clusters stopped replacing capacity that had been cycled out during routine maintenance. This caused a gradual reduction in available capacity over several hours. A subsequent deployment was activated with insufficient capacity, causing all new requests for that compute cluster to fail. To mitigate the impact, we reverted to a previous release revision, which still had sufficient capacity.

Impact: For approximately 30 minutes, customers whose traffic was handled by this compute cluster saw full downtime; other customers were unaffected. No customer data was lost.

Moving forward: We have added additional monitoring to detect this type of capacity-scaling failure much earlier, and have added safeguards to prevent deployments from shifting traffic before sufficient healthy capacity is confirmed.

_Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average._