403 when pushing to newly-created project for a while

Summary

After creating a new project, when I attempt to push to it for the first several minutes of its life, I get a 403. After some amount of time, this clears up and I can push normally from then on. I suspect this is because permissions are being populated.

Steps to reproduce

I have a very large install which one group ("main") which has at least 14k subprojects (and I think maybe more like 40k -- not sure how to tell) under various nested subgroups. The main group has as developers ~1.5k users.

When I create a new project (which I do through the API), and then try to push to that project, I get a 403. If I wait and retry and change nothing, I eventually get a success. Most recently it took me more than 17 minutes after creating a project to be able to push.

This is not new -- it's been this way for a while, but as our number of users and projects have grown, the slowdown has gotten more and more annoying.

What is the expected correct behavior?

I would expect to be able to push to a project as soon as the API request to create it finishes.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

System information
System:		Debian 9.redacted
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.27.0
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	13.2.8-ee
Revision:	593e80a4ee2
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		https://redacted.example.com
HTTP Clone URL:	https://redacted.example.com/some-group/some-project.git
SSH Clone URL:	git@redacted.example.com:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: kerberos_spnego

Results of GitLab application Check

Expand for output related to the GitLab application check 
# sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.3.0 ? ... OK (13.3.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... no
Try fixing it:
sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ;
sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
126/6 ... yes
... ~40k lines skipped, all saying "yes" ...
9010/43471 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.22.0 ? ... yes (2.27.0)
Git user has default SSH configuration? ... no
Try fixing it:
mkdir ~/gitlab-check-backup-1603471788
sudo mv /var/opt/gitlab/.ssh/id_rsa.pub ~/gitlab-check-backup-1603471788
sudo mv /var/opt/gitlab/.ssh/id_rsa ~/gitlab-check-backup-1603471788
sudo mv /var/opt/gitlab/.ssh/authorized_keys.bak ~/gitlab-check-backup-1603471788
For more information see:
doc/ssh/README.md in section "SSH on the GitLab server"
Please fix the error above and rerun the checks.
Active users: ... 1622
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... no
Try fixing it:
Please migrate all projects to hashed storage
as legacy storage is deprecated in 13.0 and support will be removed in 14.0.
For more information see:
doc/administration/repository_storage_types.md
Elasticsearch version 6.x - 7.x? ... skipped (elasticsearch is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
Edited by 🤖 GitLab Bot 🤖