Project import from template has large failure rate during mass simultaneous imports

Summary

During GitLab training workshops, we have had between 60% and 95% failure rates when importing a project from a template. This process was recently introduced in our training workshops so it is unknown if this worked previously.

This issue is to identify the root cause of the problem. Contributing factors:

This action is being performed simultaneously by many users (+/- 10 seconds) so it is possible this is a CPU/memory/thread/worker load problem that we need to diagnose.
- 185 Simultaneous Users - Est 95% Failure Rate - 2021-01-13 Advanced CI/CD Workshop - US East
- 92 Simultaneous Users - Est 66% Failure Rate - 2021-01-12 GitLab Security Workshop - Public Sector
There are no rate limit configurations that are known for this workflow, however they may exist behind the scenes and need to be identified.
A project used as a template may have a configuration or historical commit/MR/etc that causes problem.
One of the project templates that was re-created as a fresh project with a single initial commit of the source code and no other configurations imports successfully during preliminary triage.
This issue has been reproduced in different groups with different sample projects.

Steps to reproduce

User Experience

The steps that the students follow in the workshop can be found in this slide deck. (GitLab internal).
https://docs.google.com/presentation/d/1GCNIc3rKjcnIpbvv6kBBa3ekDYHmMt_Ha1xPdR6FQ4s/edit?usp=sharing

The invitation codes are time based. Please contact @jeffersonmartin on Slack to request a new code if needed.

Admin Access

Please use the handbook instructions for task 1 and 2 of the Creating and Accessing your Account tutorial. This will grant you admin access to the Demo Systems.

For shell access, please schedule a Zoom call with @jeffersonmartin to perform a screenshare or via Slack/issue comment to request log exports.

Example Project

The projects that fail:

Advanced CI Workshop - 7.2 MB - https://gitlab-core.us.gitlabdemo.cloud/training-users/session-7371fbc5/sample-projects-7371fbc5/advanced-ci-cd-workshop-jan-2021 - Export project button did not give a download link, so a problem under the hood
Security Workshop Project 1 - 7.2MB - https://gitlab-core.us.gitlabdemo.cloud/training-users/session-ed3f82e1/sample-projects-ed3f82e1/workshop-project-1 - 2021-01-13_16-42-010_training-users_session-ed3f82e_export.tar.gz
Security Workshop Project 2 - 7.1MB - https://gitlab-core.us.gitlabdemo.cloud/training-users/session-ed3f82e1/sample-projects-ed3f82e1/workshop-project-2 - 2021-01-13_16-43-808_training-users_session-ed3f82e_export.tar.gz

The simplified project that succeeds:

Advanced CI Workshop - 566 KB - https://gitlab-core.us.gitlabdemo.cloud/training-users/session-7371fbc5/sample-projects-7371fbc5/advanced-ci-cd-workshop-jan-2021 - Export 2021-01-13_16-13-219_training-users_session-7371fbc_export.tar.gz

What is the current bug behavior?

After clicking green Use Template button, the progress wheel spins indefinitely (15+ minutes). Various error messages appear in the logs.

When running an export of the project and refreshing to find the Download button, this error appears. Is there a rate limit defined somewhere?

What is the expected correct behavior?

The project will be imported in less than 60 seconds.

Relevant logs and/or screenshots

The relevant logs will be in the /training-users/session-7371fbc5 namespace and child groups.

The /var/gitlab/gitlab-rails/* logs can be accessed in this ZIP archive that's stored in G-Drive for security purposes. https://drive.google.com/file/d/1GNwHXFHG-Ntm6e8l1nlzChP1ZCDwGoPD/view?usp=sharing (GitLab Internal)

Advanced CI Workshop

Security Workshop

Output of checks

GCP Instance Configuration

n1-highmem-16 (16 vCPU, 104GB Memory, 200GB Storage)

Results of GitLab environment info

Expand for output related to GitLab environment info

gitlab-rake gitlab:env:info

System information
System:		Ubuntu 18.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.7.2p137
Gem Version:	3.1.4
Bundler Version:2.1.4
Rake Version:	13.0.1
Redis Version:	5.0.9
Git Version:	2.29.0
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	13.7.3-ee
Revision:	925686b2c52
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.9
URL:		https://gitlab-core.us.gitlabdemo.cloud
HTTP Clone URL:	https://gitlab-core.us.gitlabdemo.cloud/some-group/some-project.git
SSH Clone URL:	git@gitlab-core.us.gitlabdemo.cloud:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	13.14.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

gitlab-rake gitlab:check SANITIZE=true

Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 13.14.0 ? ... OK (13.14.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 33/4 ... yes [TRUNCATED. ALL show yes result] Redis version >= 4.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.2) Git version >= 2.29.0 ? ... yes (2.29.0) Git user has default SSH configuration? ... yes Active users: ... 988 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... yes (7.8.1) Checking GitLab App ... Finished Checking GitLab subtasks ... Finished

Possible fixes

cc @dennis @lmcandrew @jkrooswyk @lmwilliams @juliebyrne @reshmikrishna @jfullam @kgoossens @mjbatgitlab @brianwald @jsandlin @kkwentus1 @disastor @rachel_hill @rhancock @dsakamoto