Geo fails to replicate avatars when using diverging storage directories
I was on a call with a customer where a bunch of uploads were failing to sync.
There were about ~1700 uploads failing to sync, and with the following piece of code on the Rails console we've found all the failures are avatars:
Geo::FileRegistry.failed.group(:file_type).count
The paths they have configured on the primary and the secondary for the local storage of uploads differs. On the primary they using something like NFS to offload the uploads, on the secondary they are using defaults.
Relevant gitlab.rb
excerpt from the primary:
gitlab_rails['shared_path'] = "/REDACTED/shared"
# ...
gitlab_rails['uploads_directory'] = "/REDACTED/uploads"
# ...
user['home'] = "/REDACTED/home"
Relevant gitlab.rb
excerpt from the secondary:
# gitlab_rails['shared_path'] = '/var/opt/gitlab/gitlab-rails/shared'
# ...
# gitlab_rails['uploads_directory'] = "/var/opt/gitlab/gitlab-rails/uploads"
# ...
# user['home'] = "/var/opt/gitlab"
Error from geo.log
:
{
"severity":"ERROR",
"time":"2019-12-09T13:29:44.813Z",
"correlation_id": "e3d24203-4086-4ead-b08b-623d1723b7fc",
"class":"Gitlab::Geo::Replication::FileTransfer",
"message":"Error transferring file",
"job_id":"3f0e4ad973b5e0caea81d939",
"error":"{:error=\u003e#\u003cHTTP::Error: Unknown MIME type: text/plain\u003e,:file_type=\u003e:avatar, :file_id=\u003e38900, :url=\u003e\"https://gitlab.<domain>/api/v4/geo/transfers/avatar/38900\"}"
}
The customer tried to download the file manually from the cli locally and that succeeded. So it's not clear why Geo is getting 404
error.
Customer issues
- https://gitlab.zendesk.com/agent/tickets/137542 & https://gitlab.zendesk.com/agent/tickets/143471: They first reported this issue. On a call we confirmed the failure only happens on avatars. In a the latter issue they let us know they don't have Geo active at the moment.
- https://gitlab.zendesk.com/agent/tickets/143335: 2.7% of Uploads unreplicated, probably also only avatars
- Geo team support request: https://gitlab.com/gitlab-com/geo-customers/issues/25
Edited by Toon Claes