Docs feedback: Deploy tokens not replicated among HA setup

Summary

I'm using Gitlab-ee 11.10.4 and gitlab runner 11.10.1-1 in Basic HA Configuration: https://docs.gitlab.com/ee/administration/high_availability/README.html#basic-scaling

Additionally, Uploads, LFS, Artifacts and Backups are configured to their own Buckets using Object Storage configuration.

Are these deploy tokens stored in a place where the Gitlab Application servers can access them without storing them in one particular server?

Steps to reproduce

Setup self hosted Gitlab in HA scaling. Create a deploy token for a repository. Use that token to run a CI Job.

Example Project

This occurs in Gitlab Self hosted. Not sure if it occurs in gitlab.com (If you are using an older version of GitLab, this will also determine whether the bug has been fixed in a more recent version)

What is the current bug behavior?

Few times I see 401 response in production_json.log and 500 response in Gitlab Runner. Other times it passes on flawless.

What is the expected correct behavior?

(What you should see instead)

Relevant logs and/or screenshots

Gitlab-Runner Logs:

fatal: Unable to access 'https://gitlab-ci-token:[MASKED]@gitlab.example.com/GROUP/REPO.git/': The requested URL returned error: 500

ERROR: Job failed: exit status 1

production_json.log Logs:

{"method":"GET","path":"/GROUP/REPO.git/info/refs","format":"*/*","controller":"Projects::GitHttpController","action":"info_refs","status":401,"duration":38.97,"view":9.42,"db":5.02,"time":"2019-05-17T11:31:23.238Z","params":[{"key":"service","value":"git-upload-pack"},{"key":"namespace_id","value":"GROUP"},{"key":"project_id","value":"REPO.git"}],"remote_ip":"x.x.x.x","user_id":null,"username":null,"ua":"git/2.9.3","queue_duration":null,"correlation_id":"a0d231ee-c730-4621-9703-fe6b52a5cf4f"}

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)