Skip to content

Geo fails to replicate avatars when using diverging storage directories

I was on a call with a customer where a bunch of uploads were failing to sync.

There were about ~1700 uploads failing to sync, and with the following piece of code on the Rails console we've found all the failures are avatars:

Geo::FileRegistry.failed.group(:file_type).count

The paths they have configured on the primary and the secondary for the local storage of uploads differs. On the primary they using something like NFS to offload the uploads, on the secondary they are using defaults.

Relevant gitlab.rb excerpt from the primary:

gitlab_rails['shared_path'] = "/REDACTED/shared"
# ...
gitlab_rails['uploads_directory'] = "/REDACTED/uploads"
# ...
user['home'] = "/REDACTED/home"

Relevant gitlab.rb excerpt from the secondary:

# gitlab_rails['shared_path'] = '/var/opt/gitlab/gitlab-rails/shared'
# ...
# gitlab_rails['uploads_directory'] = "/var/opt/gitlab/gitlab-rails/uploads"
# ...
# user['home'] = "/var/opt/gitlab"

Error from geo.log:

{
    "severity":"ERROR",
    "time":"2019-12-09T13:29:44.813Z",
    "correlation_id": "e3d24203-4086-4ead-b08b-623d1723b7fc",
    "class":"Gitlab::Geo::Replication::FileTransfer",
    "message":"Error transferring file",
    "job_id":"3f0e4ad973b5e0caea81d939",
    "error":"{:error=\u003e#\u003cHTTP::Error: Unknown MIME type: text/plain\u003e,:file_type=\u003e:avatar, :file_id=\u003e38900, :url=\u003e\"https://gitlab.<domain>/api/v4/geo/transfers/avatar/38900\"}"
}

The customer tried to download the file manually from the cli locally and that succeeded. So it's not clear why Geo is getting 404 error.

Customer issues

Edited by Toon Claes