Per-commit checks in ChangeAccess are too slow

Summary

Pushes time out with internal API unreachable due to Checks::ChangeAccess#commits_check being too slow.

This appears to be caused by us doing a raw git lookup for every single new commit in the push.

Steps to reproduce

1. Clone gitlab-org/gitlab-ce repo
2. Create new project on GitLab.com or local dev instance and add as remote
3. Enable a per commit check, such as file locking or file name regex checks.
4. Push master branch to new project
5. This fails with GitLab: API is not accessible

remote: Resolving deltas: 100% (490366/490366), done.
remote: Checking connectivity: 629646, done.
remote: GitLab: API is not accessible
To gitlab.com:jamedjo/test-ce.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'git@gitlab.com:jamedjo/test-ce.git'

Locally additional logging can be added in Repository#new_commits to spot that the timeout happens there:

    logger = Logger.new("#{Rails.root}/log/change_access.log")
    logger.info("COMMIT rev_list: #{sha}")

My local machine was able to process around 364 commits in the 60 seconds, or around 160ms per commit. This is particularly troubling when trying to do a first push into a project, where we might expect ~10,000 commits.

Possible fixes

• ✅ We might be able to avoid entirely when there are no LFS file locking rules, and no EE Gitlab File Locking rules. I think this was previously guarded by return if deletion? || !(commit_validation || validate_path_locks?) before https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4091
• ✅ I also think we call repository.new_commits twice at the moment. Once from commits_check and once additionally in EE from push_rule_branch_check. We might be able to speed things up by re-using the list of commits.
• We might be able to fail earlier by using lazy evaluation instead of loading all commits before we start checking them
• Possibly only checking a limited number of commits?
• Explore moving the file locking check to post receive, similar to https://gitlab.com/gitlab-org/gitlab-ce/issues/42905
• Re-use the result of previously computed checks as mentioned below in https://gitlab.com/gitlab-org/gitlab-ce/issues/43281#note_71110285 and in https://gitlab.com/gitlab-org/gitlab-ce/issues/44679
• Allow processing to take more than 60 seconds somehow.

Related

• https://gitlab.com/gitlab-org/gitlab-ce/issues/41141
• https://gitlab.com/gitlab-com/support-forum/issues/3047
• https://gitlab.com/gitlab-org/gitlab-ce/issues/42905
• https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4091#note_57586481