Frequently blocked during deploys (Persisted in redis, but not rack attack??)

Summary

My account often gets blocked 403 when I am deploying multiple services in a short time period. This has been happening for months, likely a year plus, most of the time its fine for me to look around for a bit and fix the issue with restarts or just by waiting, but I figured I should write an issue.

When trying to fix this, I find myself at https://docs.gitlab.com/ee/security/rack_attack.html#remove-blocked-ips-from-rack-attack-via-redis but this never shows that I have been banned. I can add my IP (and the runners IP) to the rack attack whitelist and it will still happen. I can even disable the rack attack and it will still happen. I have learnt that flushing the whole redis db will allow me back in, so I suspect the command in the docs above is out of date?

I see that both the gitlab registry and sentry both have issues requesting during this blocked period, so I assume they are using a key that is attached to my now blocked profile.

Would love to find a way to completely disable all of this. Deploying multiple services at once guarantees a number of machines fail and its a real pain to recover from.

Steps to reproduce

Self-hosted GitLab (12.9.2-ee currently but has happened for months) Use GitLab container registry Use Sentry releases Use self-hosted GitLab CI runner limited to 2 jobs at a time

Example Project

Private self-hosted

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info



System information
System:         Ubuntu 18.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.6.5p114
Gem Version:    2.7.10
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  5.0.7
Git Version:    2.24.1
Sidekiq Version:5.2.7
Go Version:     unknown

GitLab information
Version:        12.9.2-ee
Revision:       0ad76f4d374
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     10.12
URL:            redacted
HTTP Clone URL: redacted/some-group/some-project.git
SSH Clone URL:  git@gitlab.redacted:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: google_oauth2

GitLab Shell
Version:        12.0.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 12.0.0 ? ... OK (12.0.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ...
piedpiper / fl-client ... yes
piedpiper / server ... yes
not-aviato / kraken ... yes
dl / privateemail-extension ... yes
dl / zendesk-export ... yes
CMNG / FL ... yes
dl / mono ... yes
piedpiper / dl-client ... yes
piedpiper / Payment Proxy ... yes
piedpiper / automation ... yes
avtk / organisation ... yes
Noah / Noah API ... yes
Noah / Noah Mobile ... yes
Noah / Noah Nest Api ... yes
Noah / Noah Web Client ... yes
£ / stella ... yes
piedpiper / elk ... yes
piedpiper / qa ... yes
opencontracts / design ... yes
opencontracts / api ... yes
opencontracts / ui ... yes
inkage / design ... yes
inkage / api ... yes
inkage / ui ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.5)
Git version >= 2.22.0 ? ... yes (2.24.1)
Git user has default SSH configuration? ... yes
Active users: ... 2
Is authorized keys file accessible? ... yes
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

root@gitlab:#
root@gitlab:#
root@gitlab:~# sudo gitlab-rake gitlab:check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 12.0.0 ? ... OK (12.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... piedpiper / fl-client ... yes piedpiper / server ... yes not-aviato / kraken ... yes dl / privateemail-extension ... yes dl / zendesk-export ... yes CMNG / FL ... yes dl / mono ... yes piedpiper / dl-client ... yes piedpiper / Payment Proxy ... yes piedpiper / automation ... yes avtk / organisation ... yes Noah / Noah API ... yes Noah / Noah Mobile ... yes Noah / Noah Nest Api ... yes Noah / Noah Web Client ... yes £ / stella ... yes piedpiper / elk ... yes piedpiper / qa ... yes opencontracts / design ... yes opencontracts / api ... yes opencontracts / ui ... yes inkage / design ... yes inkage / api ... yes inkage / ui ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.24.1) Git user has default SSH configuration? ... yes Active users: ... 2 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

redis> flushall 🔥

Edited Apr 25, 2020 by Ghost User