Elevated errors when decrypting ESC environments

Some Pulumi Cloud organizations experienced elevated errors when decrypting ESC environments. A subset of environment revisions were affected, and those revisions failed to decrypt 100% of the time during the window. Overall, 3.32% of decrypt-attempt requests across Pulumi Cloud failed due to this issue.

At the start of the window, we rolled out a change that prevented secret ciphertext copied from an ESC environment from being decrypted in other environments. Customers who had previously copied ciphertext between environments were affected. Users saw a message: “decryption failed. If this secret's ciphertext was copied from a different environment, copy and paste the plaintext instead.” This change was intended as a security improvement, but the customer impact was higher than expected, so the change was rolled back.

The ESC team has conducted an internal postmortem to understand the misjudged customer impact, improve our responsiveness to scenarios like this, and created a plan to launch this security improvement without customer disruption.