Skip to content

Pass around GIT_*_OBJECT_DIR* variables as relative paths

Related

Git (git-receive-pack) sets GIT_ALTERNATE_OBJECT_DIRECTORIES and GIT_OBJECT_DIRECTORY to absolute paths. This is not compatible with Gitaly. When passing these values, from Git hooks to the GitLab API, or in Gitaly::Repository messages, we must use relative paths.

Also take extra care with the fact that GIT_ALTERNATE_OBJECT_DIRECTORIES is a list. Don't presume it always has just one entry.

Reference: #627 (closed)

Reference: #525 (closed)

@jacobvosmaer-gitlab wrote:

The combined rails logs from the API in my test environment paint a clear picture though:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/api/v4/internal/allowed" for 172.18.0.2 at 2017-10-04 12:04:20 +0000

==> /var/log/gitlab/gitlab-rails/api_json.log <==
{"time":"2017-10-04T12:04:20.257Z","severity":"INFO","duration":96.27,"db":4.74,"view":91.53,"status":500,"method":"POST","path":"/api/v4/internal/allowed","params":{"action":"git-receive-pack","changes":"19df4a8a875e7ebe0fb70b97080c75b89a7ac440 95640b336a8f2ae4aae68ab2cd4b8a087e814d7c refs/heads/master\n","gl_repository":"project-1","project":"/var/opt/gitlab/git-data/repositories/root/lalala.git","protocol":"http","env":"{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":\"/var/opt/gitlab/git-data/repositories/root/lalala.git/./objects\",\"GIT_OBJECT_DIRECTORY\":\"/var/opt/gitlab/git-data/repositories/root/lalala.git/./objects/incoming-lGxxiF\"}","user_id":"1","secret_token":"[FILTERED]"},"host":"app1","ip":"172.18.0.2","ua":"Ruby"}

==> /var/log/gitlab/gitlab-rails/production.log <==

RuntimeError (Got a non-zero exit code while calling out `/opt/gitlab/embedded/bin/git --git-dir=/mnt/data1/repositories/root/lalala.git rev-list --max-count=1 19df4a8a875e7ebe0fb70b97080c75b89a7ac440 ^95640b336a8f2ae4aae68ab2cd4b8a087e814d7c`.):
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/rev_list.rb:32:in `execute'
  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/rev_list.rb:23:in `missed_ref'

Note how the API receives GIT_ALTERNATE_OBJECT_DIRECTORIES and GIT_OBJECT_DIRECTORY values starting with /var/opt/gitlab/git-data/repositories, and how the failed command runs in /mnt/data1/repositories.

Also note that the exception trace matches the exceptions we saw in production during the trial. https://sentry.gitlap.com/gitlab/gitlabcom/issues/48184/

The difficult thing is we have all sorts of 500 errors on this API, now that I look for it https://log.gitlap.com/goto/1b11b68af2eefdd25dbcc15dbf79cac2 .

I had to find the right sentry query: https://sentry.gitlap.com/gitlab/gitlabcom/?query=is%3Aunresolved+url%3Ahttp%3A%2F%2F0.0.0.0%3A8080%2Fapi%2Fv4%2Finternal%2Fallowed

Edit: but, we are definitely getting new internal API errors when pushing to a protected branch via gitaly. These errors do not occur with the gitaly push features disabled.

Edited by Andrew Newdigate
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information