DeadlineExceeded for large, non-LFS files in an MR can block "policy rules must be satisfied" check

Summary

In this internal customer report, the error message Something went wrong on our end on the Changes tab on an MR with many binary files, and under Checking if merge request can be merged, All policy rules must be satisfied never completed loading.

However, it was not clear why the problem occured. Logs showed grpc_status:4, grpc_message:"Deadline Exceeded" (ca 10.sec, i.e. the fast Gitaly timeout, but this should be surfaced to the user.

Steps to reproduce

  1. On a test instance (v18.7.2), create a non-admin user, a group, and a test project.
  2. Fill the test project with a bunch of non-LFS but binary files. I used this tool 
gls_generate_commits --bytes=64M --number=12
gls_generate_commits --bytes=48M --number=12
gls_generate_commits --bytes=24M --number=12
gls_generate_commits --bytes=12M --number=12
gls_generate_commits --bytes=1M --number=12
git push -u origin main

# results in ca. 5GB large repo, probably larger than necessary
  1. (optional) Reduce Gitaly timeouts to speed up the testing.
  2. Create a diff in those binary files and a text file, a new branch and an MR.
date > README.md
for f in $(/bin/ls 12x-*/* | shuf); do echo "$(date +%s)" >> "$f" ; done
  1. Observe that the MR diff does not load, banner Something went wrong on our end. Please try again. appears and logs show GRPC::DeadlineExceeded, but merging is possible.
  2. Create the attached approval policy in the group, and repeat step 3, creating a new MR with the non-admin user. approval_policy.yaml
  3. This time, Something went wrong on our end. is associated with endless spinning of All policy rules must be satisfied, which blocks merging.

image

Workaround: Migrating all binary files to LFS resolved the problem.

Interestingly, I was not able to reproduce this, when the admin user was the MR author 🤔

Example Project

What is the current bug behavior?

See step 6. ⬆️

What is the expected correct behavior?

  1. Policy rules retrieval and check should not be affected by DeadlineExceeded for the MR's changed files.
  2. Similarly to LFS files and very large diffs, the Changes tab should flag them as not displayable, and suggest .diff downloading or reviewing local with git diff or similar, but load the text diffs fine.
  3. The Something went wrong on our end message should include back-end details, for searching known issues, GitLab forum, etc. or even pointer to the solutions /help/topics/git/lfs/#migrate-an-existing-repository-to-git-lfs or /help/administration/settings/gitaly_timeouts/#available-call-timeouts. In particular, I find Please try again misleading, because it's not clear why a repetition without any changes would yield a successful policy rules check.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

# gitlab-rake gitlab:env:info

System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   3.2.8
Gem Version:    3.7.1
Bundler Version:2.7.1
Rake Version:   13.0.6
Redis Version:  7.2.11
Sidekiq Version:7.3.9
Go Version:     unknown

GitLab information
Version:        18.7.2-ee
Revision:       cb985c3c109
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     16.10
URL:            https://gdk.test
HTTP Clone URL: https://gdk.test/some-group/some-project.git
SSH Clone URL:  ssh://git@gdk.test:2222/some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        14.45.5
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      18.7.2
- default Git Version:  2.50.1

Results of GitLab application Check

Expand for output related to the GitLab application check 
gitlab-rake gitlab:check SANITIZE=true



Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 14.45.5 ? ... OK (14.45.5)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
2/2 ... yes
Redis version >= 6.2.14? ... yes
Ruby version >= 3.0.6 ? ... yes (3.2.8)
Git user has default SSH configuration? ... yes
Active users: ... 2
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-9.x or OpenSearch version 1.x-3.x ... skipped (advanced search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.

Edited by Katrin Leinweber