Skip to content

Incident detail

Authentication issues related to API requests

Resolved incidentCritical2 affected services

Timeline window

to

Outage alerts

Get alerted the next time GitHub breaks

Free email alerts for up to 5 providers. No card, live in about a minute. Paid plans add Slack, Teams, Discord, and webhook delivery across your whole stack, plus higher API quotas.

Timeline

Incident updates

Every update GitHub posted, oldest to newest, exactly as it appeared on their official status page.

  1. Investigating

    We are investigating reports of impacted performance for some GitHub services.

  2. Investigating

    API Requests is experiencing degraded availability. We are continuing to investigate.

  3. Investigating

    Issues is experiencing degraded performance. We are continuing to investigate.

  4. Investigating

    We are investigating issues related to sporadic authentication failures impacting approximately 15% of API traffic. We will continue to investigate and provide updates.

  5. Investigating

    We continue to investigate issues related to sporadic authentication failures, impacting approximately 15% of API traffic. Further updates will be provided as we work to mitigate.

  6. Investigating

    The degradation affecting Issues has been mitigated. We are monitoring to ensure stability.

  7. Investigating

    We continue to investigate issues related to sporadic authentication failures, impacting approximately 15% of API traffic. Erroneous 401 responses are causing app integrations to trigger authentication flows. We have identified a problematic component in our infrastructure and are working to mitigate.

  8. Investigating

    The degradation affecting API Requests has been mitigated. We are monitoring to ensure stability.

  9. Monitoring

    The degradation has been mitigated. We are monitoring to ensure stability.

  10. Resolved

    Between 15:05 UTC and 16:25 UTC, GitHub API services experienced degraded availability due to sporadic authentication failures affecting approximately 9% of requests. Customers experienced intermittent "logged out" behavior as erroneous 401 responses triggered repeated authentication flows in app integrations. Affected requests also experienced approximately 800ms of additional latency. <br /><br />A memcached proxy service rollout to our internal API infrastructure caused our authentication service to pick up an incorrect memcached host configuration, leading to intermittent authentication lookup failures. We mitigated the incident by deploying a configuration change to memcached to use the correct host. <br /><br />To prevent similar issues in the future, we plan to migrate our authentication system to the new memcached infrastructure to improve resilience and strengthen overall reliability posture.

Keep exploring

More from GitHub

Neighboring incidents on GitHub's timeline and the rest of their record on OutageDeck.