{"meta":{"version":"v1","pricing":{"public":{"label":"Public","description":"Read-only API access for lightweight status checks and public integrations."},"premium":{"label":"Premium (early access)","description":"API keys with higher hourly quotas are available now; webhooks, alerting, and exports are on the roadmap."}},"generatedAt":"2026-07-08T22:19:03.911Z"},"data":{"id":"incident_statuspage_pypi_77qzc7qsb5wb","slug":"pypi-pypi-backends-are-unavailable-2022-10-24","title":"PyPI Backends are unavailable.","summary":"PyPI Backends are unavailable.","status":"resolved","severity":"major","startedAt":"2022-10-24T14:51:38.946+00:00","updatedAt":"2022-10-24T16:05:22.902+00:00","resolvedAt":"2022-10-24T15:44:39.73+00:00","provider":{"slug":"pypi","name":"PyPI"},"affectedServices":[{"slug":"pypi-files","name":"Package Downloads"},{"slug":"pypi-index","name":"PyPI Package Index"}],"links":{"html":"/incidents/pypi-pypi-backends-are-unavailable-2022-10-24","api":"/api/v1/incidents/pypi-pypi-backends-are-unavailable-2022-10-24","providerHtml":"/providers/pypi"},"impactSummary":"PyPI reported a major event for the affected tracked services.","source":{"id":"source_pypi_status","kind":"official_api","name":"Python Infrastructure Status","checkedAt":"2026-07-08T22:15:02.283+00:00","officialUrl":"https://status.python.org","statusPageUrl":"https://status.python.org"},"updates":[{"id":"update_statuspage_pypi_77qzc7qsb5wb_qrcfw0k8q66x","status":"investigating","body":"The backend that hosts PyPI and associated services is experiencing a major outage, we are investigating.","createdAt":"2022-10-24T14:51:39.025+00:00"},{"id":"update_statuspage_pypi_77qzc7qsb5wb_y8p0q6rfyk9v","status":"monitoring","body":"We have identified and resolved the reason for the outage and are monitoring to ensure it remains stable.","createdAt":"2022-10-24T15:05:05.438+00:00"},{"id":"update_statuspage_pypi_77qzc7qsb5wb_yfnsf6ym5br2","status":"resolved","body":"This incident has been resolved.","createdAt":"2022-10-24T15:44:39.73+00:00"},{"id":"update_statuspage_pypi_77qzc7qsb5wb_k2zbdb2dkz9p","status":"resolved","body":"## Summary\n\nThe cluster that hosts PyPI’s backends as well as multiple ancillary services experienced an outage during maintenance that interrupted access to services over HTTPS.\n\n## Details\n\nFrom 2022-10-24 14:43 UTC until 2022-10-24 15:03 UTC, PyPI’s backends were not accessible over HTTPS. This interfered with our CDNs ability to fetch pages, made uploads to [uploads.pypi.org](http://uploads.pypi.org) impossible, and interrupted other services such as our legacy file redirect service.  \n  \nPyPI services run as deployments in a Kubernetes cluster and are exposed via Ingress with the AWS Elastic Load Balancer \\(ELB\\) integration. The TLS certificate that this load balancer uses is managed via Amazon’s Certificate Manager \\(ACM\\). When initially deploying our Kubernetes cluster, the Ingress managed ELB was configured to use an existing ACM TLS certificate.  \n  \nEarlier today, regular maintenance of our Kubernetes cluster required a rolling restart of all nodes in the cluster to distribute upgrades and new configurations to all nodes. During this rolling restart as the Kubernetes API server hosts were deployed, Kubernetes validated and refreshed the Ingress configurations we had previously defined.  \n  \nSince the most recent rolling upgrade, an additional hostname was needed for PyPI’s Ingress. PyPI administrators created a new ACM TLS certificate including that hostname and updated the Ingress managed ELB to use this new certificate. As a result, the new Kubernetes API servers were unable to find the previous ACM TLS certificate and disabled the HTTPS listener for the Ingress configuration that serves PyPI and associated services as they came online.  \n  \nOnce identified, the PyPI admins updated the Ingress configuration to point to the new ACM TLS certificate and Kubernetes restored the HTTPS listener on the Ingress managed ELB, restoring access to all services.\n\n## Mitigation\n\nWe will investigate mechanisms by which the hostnames needed on ACM TLS certificates for PyPI’s Ingress configurations can be managed via Kubernetes resources rather than manually via the AWS console. By managing resources all via the Kubernetes API, drift between desired state and reality will be less likely to occur and surface during similar maintenance in the future.","createdAt":"2022-10-24T15:44:50.488+00:00"}],"access":{"plan":"public","keyed":false}}}