Issues with RackAttack rate-limiting of "go get"
It looks like our RackAttack rate-limiting will affect go get
in at least some scenarios. There are a number of oddities with the traffic pattern in question that need an explanation, and we need to consider what mitigations (if any) we need to implement, to avoid causing undue problems for our customers.
A useful log search is https://log.gprd.gitlab.net/goto/41845ac549415f3447f4f734a4a5ac48 which shows one recent occurrence (2020-12-06, so we don't have long to review this particular one in detail). Anecdotally while doing the RackAttack log analysis I saw other similar occurrences, although this is a particularly substantial one.
The oddities:
- Of the 41809 log entries, 5286 were actual Rails request logs, and the remaining 36523 were from RackAttack. This is weird because:
- I thought we could only have one throttle apply per Rails request, so if every single request were rate-limited we should have a 1:1 ratio (throttle:rails), instead we have a 7:1 ratio.
- Correlation ID's do not line up, e.g. when the rate-limiting kicks in, we see
track
logs with acorrelation_id
that has no related Rails log. There are some (e.g. correlation id 01ERWBM6TXXBS3QDVTX804QK37), but they seem rare.
- It looks like
go get
has some sort search function (addjson.ua = Go-http-client/2.0
to the query), where it truncates the repo path character by character, and sometimes ends up querying/
. This could be a real problem for users, with the client code spamming us with checks for non-existent paths that count against the limit.
Out of scope: Lots of 302 redirects to sign_in; but I think this is because the repos are private and this was unauthenticated traffic, so the fact that this was 'failing' for the customer in question is not a problem we have to solve.
The logging oddness needs an explanation and perhaps a fix depending on what we find.
What we do in general about the behavior of go get
is an open question. We certainly can't just bypass for a certain User-Agent + "go-get=1" in the query string (entirely too easily abused), but I'm not sure what else we can do. Maybe nothing, and just make sure Support know?