GitLab::RackAttack: detect basic authentication with git/lfs
Summary
GitLab::RackAttack is used to throttle and block users and IP addresses:
- For unauthenticated requests, throttling is done per IP address.
- For authenticated requests, throttling is done per user.
That's fine. But there's a bug with git access (e.g. git clone
or Git LFS downloads) using Basic Authentication
: On such requests, GitLab::RackAttack does not detect authentication, even though it's supplied using Basic Authentication
. The consequence is that the request is falsely recognized as unauthenticated and therefore the IP address is throttled (or blocked) instead of the user. So when an IP address is throttled because of an abusive user, all other clients originating from the same shared IP address will be blocked temporarily. We often encounter this problem in connection with CI runners.
Steps to reproduce
- Create a project with a repository
- Enable Git LFS and add some files to the repository
- Run
git clone
and debug the methodGitlab::RackAttack::Request#unauthenticated?
What is the current bug behavior?
Let's run git clone
on a project using LFS:
Request by git
:
Note: "By default, all Git operations are first tried unauthenticated." (see docs)
First, the following request is made:
Request: GET /user1/p2.git/info/refs/?service=git-upload-pack
HTTP_ACCEPT: */*
HTTP_USER_AGENT: git/2.31.1
HTTP_AUTHORIZATION:
==> Gitlab::RackAttack::Request#unauthenticated? = true
That's correct, no authentication was included.
As the previous request requires authentication, the same request is repeated, but including Basic Authentication
:
Request: GET /user1/p2.git/info/refs/?service=git-upload-pack
HTTP_ACCEPT: */*
HTTP_USER_AGENT: git/2.31.1
HTTP_AUTHORIZATION: Basic dXNlcjpteV9wYXNzd29yZF9pc19ub3RfaGVyZQ==
==> Gitlab::RackAttack::Request#unauthenticated? = true
That's NOT correct, RackAttack does not recognize the included Basic Authentication
.
Request by git lfs
:
The same happens with the POST request by git-lfs:
Request: POST /user1/p2.git/info/lfs/objects/batch
HTTP_ACCEPT: application/vnd.git-lfs+json; charset=utf-8
HTTP_USER_AGENT: git-lfs/2.13.3 (GitHub; darwin amd64; go 1.16.2)
HTTP_AUTHORIZATION:
==> Gitlab::RackAttack::Request#unauthenticated? = true (correct)
Request: POST /user1/p2.git/info/lfs/objects/batch
HTTP_ACCEPT: application/vnd.git-lfs+json; charset=utf-8
HTTP_USER_AGENT: git-lfs/2.13.3 (GitHub; darwin amd64; go 1.16.2)
HTTP_AUTHORIZATION: Basic dXNlcjpteV9wYXNzd29yZF9pc19ub3RfaGVyZQ==
==> Gitlab::RackAttack::Request#unauthenticated? = true (NOT correct)
In the GET request to the LFS object, Basic Authentication
is included from the start, with the same result:
Request: GET /user1/p2.git/gitlab-lfs/objects/33778b47d1e1436ad6ae3f...
HTTP_ACCEPT:
HTTP_USER_AGENT: git-lfs/2.13.3 (GitHub; darwin amd64; go 1.16.2)
HTTP_AUTHORIZATION: Basic dXNlcjpmeUpoYkdjaU9pSklVekkxTmlJc0luUjVjQ0...
==> Gitlab::RackAttack::Request#unauthenticated? = true (NOT correct)
What is the expected correct behavior?
The method Gitlab::RackAttack::Request#unauthenticated?
recognizes the Basic Authentication
by git
and git-lfs
as valid authentication.
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
Bundler Version:2.1.4 Rake Version: 13.0.3 Redis Version: 6.0.10 Git Version: 2.31.1 Sidekiq Version:5.2.9 Go Version: go1.15.5 darwin/amd64 GitLab information Version: 13.12.0-pre Revision: 025cebff793 Directory: /Users/x/repos/gdk/gitlab DB Adapter: PostgreSQL DB Version: 11.9 URL: http://gdk.test:3000 HTTP Clone URL: http://gdk.test:3000/some-group/some-project.git SSH Clone URL: ssh://x@gdk.test:2222/some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2 GitLab Shell Version: 13.18.0 Repository storage paths: - default: /Users/x/repos/gdk/repositories GitLab Shell path: /Users/x/repos/gdk/gitlab-shell Git: /usr/local/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.18.0 ? ... OK (13.18.0) Running /Users/x/repos/gdk/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... yes Init script up-to-date? ... yes Projects have namespace: ... 22/1 ... yes 22/2 ... yes 23/3 ... yes 24/4 ... yes 25/5 ... yes 26/6 ... yes 27/7 ... yes 28/8 ... yes 11/9 ... yes 5/10 ... yes 14/11 ... yes 19/12 ... yes 3/13 ... yes 8/14 ... yes 15/15 ... yes 17/16 ... yes 9/17 ... yes 18/18 ... yes 52/19 ... yes 109/41 ... yes 110/56 ... yes 107/57 ... yes 111/58 ... yes 120/60 ... yes 120/61 ... yes 120/62 ... yes 120/63 ... yes 120/64 ... yes 120/65 ... yes 120/66 ... yes 120/67 ... yes 120/68 ... yes 120/69 ... yes 120/70 ... yes 120/71 ... yes 120/72 ... yes 120/73 ... yes 120/74 ... yes 120/75 ... yes 120/76 ... yes 120/77 ... yes 120/78 ... yes 120/79 ... yes 120/80 ... yes 120/81 ... yes 120/82 ... yes 120/83 ... yes 120/84 ... yes 132/85 ... yes 107/86 ... yes 107/87 ... yes Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.2) Git version >= 2.31.0 ? ... yes (2.31.1) Git user has default SSH configuration? ... yes Active users: ... 97 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
Inspect the unauthenticated?
method in lib/gitlab/rack_attack/request.rb
and extend the corresponding methods in lib/gitlab/auth/request_authenticator.rb
and lib/gitlab/auth/auth_finders.rb
.