Repository storage moves API fails with "both source and repository should have the same relative path"
Summary
When using the repository storage moves API to transfer projects between Gitaly storages, the Projects::ScheduleBulkRepositoryShardMovesWorker
job fails with a both source and repository should have the same relative path
error originating from Praefect.
Provided !5073 (closed) makes it into 15.6
, the severity will be severity2. Otherwise it will need to be upgraded to severity1 .
Problem is occuring on GitLab 15.5.2-ee
Steps to reproduce
- Setup a Gitaly Cluster as
default
, and a standalone Gitaly server asstorage1
- Create a project on
storage1
-
Schedule repository storage moves for all projects on a storage shard from
storage1
todefault
and see the aforementioned error in Sidekiq and Praefect. As a result, the move is not successful.
What is the current bug behavior?
The repository storage moves API does not transfer projects between storages.
What is the expected correct behavior?
The repository storage moves API should transfer projects between storages.
Relevant logs and/or screenshots
{"severity":"INFO","time":"2022-11-16T01:15:34.354Z","retry":3,"queue":"projects_update_repository_storage","version":0,"args":["260","[FILTERED]","3312"],"class":"Projects::UpdateRepositoryStorageWorker","jid":"4b77e0cfbc7a02f4f7c07209","created_at":"2022-11-16T01:15:08.986Z","correlation_id":"0cc320f3526982cbb36241750a6335b0","meta.caller_id":"Projects::ScheduleBulkRepositoryShardMovesWorker","meta.remote_ip":"136.24.66.34","meta.feature_category":"gitaly","meta.user":"gen","meta.client_id":"user/64","meta.root_caller_id":"POST /api/:version/project_repository_storage_moves","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:projects_update_repository_storage:a9b7e158c8be382cefdd84151e6680d414c9bc254d964350c9d8debdc84da4cc","size_limiter":"validated","enqueued_at":"2022-11-16T01:15:34.286Z","error_message":"3:both source and repository should have the same relative path. debug_error_string:{\"created\":\"@1668561309.066088467\",\"description\":\"Error received from peer ipv4:10.0.3.114:2305\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1063,\"grpc_message\":\"both source and repository should have the same relative path\",\"grpc_status\":3}","error_class":"ArgumentError","failed_at":"2022-11-16T01:15:09.095Z","retry_count":0,"job_size_bytes":27,"pid":621543,"message":"Projects::UpdateRepositoryStorageWorker JID-4b77e0cfbc7a02f4f7c07209: done: 0.054409 sec","job_status":"done","scheduling_latency_s":0.013412,"redis_calls":4,"redis_duration_s":0.004233,"redis_read_bytes":10,"redis_write_bytes":366,"redis_queues_calls":4,"redis_queues_duration_s":0.004233,"redis_queues_read_bytes":10,"redis_queues_write_bytes":366,"db_count":4,"db_write_count":3,"db_cached_count":0,"db_replica_count":0,"db_primary_count":4,"db_main_count":4,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.017,"db_main_duration_s":0.017,"db_main_replica_duration_s":0.0,"cpu_s":0.005108,"mem_objects":2230,"mem_bytes":163592,"mem_mallocs":389,"mem_total_bytes":252792,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.054409,"completed_at":"2022-11-16T01:15:34.354Z","load_balancing_strategy":"primary","db_duration_s":0.016848}
Possible fixes
From @proglottis (in Slack):
It's supposed to be a safety check. On the rails side, the repository on the destination storage is specified here https://gitlab.com/gitlab-org/gitlab/-/blob/8a53bb04051b00d80962ed058275b00ad61e8c39/app/services/concerns/update_repository_storage_methods.rb#[…]1 - Basically only the storage is different. So if I were to guess, this would be something to do with praefect rewriting the requests to the actual gitaly - perhaps it's the newish generated replica paths? #4218 (closed)
Workaround
Disabling the feature flag will work around the problem (run sudo gitlab-rails c
on a Rails node to apply):
Feature.disable(:gitaly_praefect_generated_replica_paths)