GitHub public token pool reaching API limit
Our GitHub public token pool is reaching API limit on GitHub's side.
(This error is coming from GitHub)
ErrorCtor: API rate limit exceeded for app ID xxx.
- https://sentry.gitlab.net/gitlab/gitter-backend/issues/651934/
- https://sentry.gitlab.net/gitlab/gitter-backend-staging/issues/741089/
The app ID xxx
aligns with the one we have configured for github__anonymous_app__client_id
, github:anonymous_app:client_id
which is used in modules/github/lib/public-token-pool.js
It's been happening occasionally over time but just had a bigger spike recently. This probably means a specific person/bot is hitting us hard vs natural growth but it still sucks that it DOS's some of the GitHub decoration and avatar stuff for others. To be clear, this isn't a huge blocker to using Gitter.
Solutions
Investigate usage
Investigate what caused the spike and stop the user(reference our ToS) or fix up the potential problem area(cache more things)
Add more public tokens
For example with GitLab, we have gitlab__public_token_pool
with 5 tokens available to rotate through
cc @viktomas