Disruption with GitHub services
Incident
Report for GitHub
Resolved
On July 02, 2024, between 18:21 UTC and 19:24 UTC the code search service was degraded and returned elevated 500 HTTP status responses. On average, the error rate was 38% of code search requests. This was due to a bad deployment causing some user's rate limit calculations to error while processing code search requests. This impacted approximately 2,000 users.
We mitigated the incident by rolling back the bad deployment along with resetting rate limits for all users.
We have identified and implemented updates in the testing of rate limit calculations to prevent this problem from happening again, and clarified deployment processes for verification before a full production rollout to minimize impact in the future.
We mitigated the incident by rolling back the bad deployment along with resetting rate limits for all users.
We have identified and implemented updates in the testing of rate limit calculations to prevent this problem from happening again, and clarified deployment processes for verification before a full production rollout to minimize impact in the future.
Update
API Requests is operating normally.
Update
The fix has been rolled out and our telemetry indicates that the errors with code search have resolved.
Update
An issue with faulty data in an in-memory storage causes around one third of code search requests to fail. The team has identified the issue and is working on rolling out a fix.
Update
API Requests is experiencing degraded performance. We are continuing to investigate.
Investigating
We are currently investigating this issue.
This incident affected: API Requests.
