Device Trust Outage

Investigating

We are aware of an outage of Device Trust and are investigating the issue.

Investigating

We are continuing to investigate the issue.

Identified

The issue has been identified and a fix is being implemented.

Resolved

A fix has been deployed and all services are operational.

Resolved

# Incident Postmortem - Device Trust Authentication Outage **Date of Incident:** 2026-07-02 **Time of Incident \(UTC\):** 19:28 - 20:28 **Service\(s\) Affected:** This incident affected: USA/Global \(Device Trust Authentication Service, Device Trust Checks Server, Device Trust Portal, Device Trust API\) and Europe \(Device Trust Authentication Service, Device Trust Checks Server, Device Trust Portal, Device Trust API\). **Impact Duration:** 1 Hour ## Summary On July 2, 2026, 1Password Device Trust experienced an outage lasting approximately one hour, beginning at 19:28 UTC. During this window, all Device Trust services were unavailable and users were unable to authenticate. We have identified the root cause: an internal configuration change to our cloud load balancing infrastructure was applied more broadly than intended which temporarily made a required endpoint unreachable. This was not a security incident and did not involve any third party. The configuration has been corrected, service has been fully restored, and we have implemented additional safeguards to help prevent this type of issue from recurring. ## Impact on Customers * **All Device Trust services were completely unavailable during the 60-minute outage window. Users were unable to authenticate to any Device Trust service, including the authentication portal \(**[**auth.kolide.com**](http://auth.kolide.com)**\), admin console \(**[**app.kolide.com**](http://app.kolide.com)**\), API endpoints, and device server.** * **Number of Affected Customers \(approximate\):** 100% of Device Trust customers * **Geographic Regions Affected \(if applicable\):** US and EU ## What Happened? On July 2, 2026, A configuration change intended to improve the security configuration of an internal service was applied more broadly than intended, affecting a shared network routing component relied on by Device Trust. The change replaced the underlying network infrastructure supporting Device Trust. However, the traffic routing configuration continued to reference the previous infrastructure instead of the newly provisioned resources. As a result, Device Trust authentication endpoints became unreachable. Reverting the change restored connectivity and service. * **Timeline of Events \(UTC\):** * 19:28: Network configuration change applied; Network Load Balancers responsible for routing traffic for Device Trust services became unreachable * 19:30: Automated monitoring detects the outage and triggers alerts * 19:33: Incident is declared at [status.1password.com](http://status.1password.com) * 20:03: Root cause is identified as a load balancer configuration issue * 20:24: Load balancer is recreated with correct configuration * 20:26 Recovery is confirmed for [app.kolide.com](http://app.kolide.com) * 20:28 Recovery is confirmed for [auth.kolide.com](http://auth.kolide.com) * **Contributing Factors \(if any\):** * **Shared component scope not visible during review or validation.** The network infrastructure class affected by the change was shared across multiple services, but this dependency was not documented or surfaced in the change review process. As a result, the pre-production validation, while performed in a lower environment before production rollout, did not cover all dependent services, and the impact on Device Trust was not identified before the change reached production. * **Concurrent diagnostic signals slowed root cause identification.** At the time the outage began, unrelated certificate renewal errors surfaced alongside the network connectivity failure. Investigating both signals simultaneously delayed identification of the network configuration change as the root cause. ## How Was It Resolved? * **Resolution Steps:** Once the source of the outage was identified, engineers reverted the configuration change. The affected network infrastructure was recreated with its original, publicly accessible configuration, restoring routing to all Device Trust authentication endpoints. Service recovery was confirmed sequentially across affected services before the incident was closed. * **Verification of Resolution:** Following the revert, engineers verified that all Device Trust authentication endpoints were reachable and accepting traffic. Customer confirmation of resolution was subsequently received through the support team. ## What We Are Doing to Prevent Future Incidents * **Require explicit approval for destructive infrastructure changes.** We are implementing safeguards that require manual review and approval before changes that would destroy and recreate critical network infrastructure can be applied. This creates a forcing function to catch unintended scope before changes reach production. * **Improve scope visibility for shared infrastructure components.** We are documenting the dependencies between shared infrastructure components and the services that rely on them, and integrating this information into our change review process. Future changes to shared components will require an explicit assessment of downstream impact before deployment. * **Strengthen pre-production validation across shared dependencies.** We are updating our change promotion process to require validation against all services that share affected components — not only the intended target — before changes are promoted to production. * **Reduce single points of failure in traffic routing.** We are evaluating architectural changes to introduce redundancy into Device Trust's traffic routing configuration, so that a failure in any single component results in degraded service rather than a complete outage. ## Next Steps and Communication * No action is required from our customers at this time. We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team