Forking a project housed on standard storage into Praefect-managed storage fails with PermissionDenied errors
Scenario
- Hosts: 1 app, 1 praefect, 1 gitaly behind praefect
Original project is created in default
storage on app server. Admin then switches storage for new projects to praefect
. When a user attempts to fork the original the import fails and no repository is created. On the gitaly node SSHUploadPack
and CreateFork
have failed with "PermissionDenied" errors.
Logs
gitaly
{"correlation_id":"4pNbjKmgOx5","error":"rpc error: code = PermissionDenied desc = permission denied","grpc.code":"PermissionDenied","grpc.meta.auth_version":"v1","grpc.method":"SSHUploadPack","grpc.service":"gitaly.SSHService","grpc.start_time":"2019-11-05T20:24:17Z","grpc.time_ms":0.168,"level":"warning","msg":"finished streaming call with code PermissionDenied","peer.address":"@","pid":13983,"span.kind":"server","system":"grpc","time":"2019-11-05T20:24:17Z"}
{"correlation_id":"4pNbjKmgOx5","error":"rpc error: code = PermissionDenied desc = permission denied","grpc.code":"PermissionDenied","grpc.meta.auth_version":"v1","grpc.method":"SSHUploadPack","grpc.service":"gitaly.SSHService","grpc.start_time":"2019-11-05T20:24:17Z","grpc.time_ms":0.119,"level":"warning","msg":"finished streaming call with code PermissionDenied","peer.address":"@","pid":13983,"span.kind":"server","system":"grpc","time":"2019-11-05T20:24:17Z"}
{"correlation_id":"4pNbjKmgOx5","grpc.meta.auth_version":"v1","grpc.meta.client_name":"gitlab-sidekiq","grpc.method":"CreateFork","grpc.request.deadline":"2019-11-06T02:25:57Z","grpc.request.fullMethod":"/gitaly.RepositoryService/CreateFork","grpc.request.glProjectPath":"whatever/parent-project","grpc.request.glRepository":"","grpc.request.repoPath":"@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git","grpc.request.repoStorage":"praefect-gitaly-1","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019-11-05T20:24:17Z","level":"error","msg":"Cloning into bare repository '/var/opt/gitlab/git-data/repositories/@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git'...\\n2019/11/05 20:24:17 upload-pack: rpc error: code = PermissionDenied desc = permission denied\\nfatal: Could not read from remote repository.\\n\\nPlease make sure you have the correct access rights\\nand the repository exists.\\n","peer.address":"10.128.0.20:46204","pid":13983,"span.kind":"server","system":"grpc","time":"2019-11-05T20:24:17Z"}
praefect
{"grpc.code":"OK","grpc.method":"SSHUploadPack","grpc.service":"gitaly.SSHService","grpc.start_time":"2019-11-05T20:30:26Z","grpc.time_ms":4.674,"level":"inf
o","msg":"finished streaming call with code OK","pid":13825,"span.kind":"server","system":"grpc","time":"2019-11-05T20:30:26Z"}
{"grpc.code":"OK","grpc.method":"SSHUploadPack","grpc.service":"gitaly.SSHService","grpc.start_time":"2019-11-05T20:30:26Z","grpc.time_ms":18.888,"level":"in
fo","msg":"finished streaming call with code OK","pid":13825,"span.kind":"server","system":"grpc","time":"2019-11-05T20:30:26Z"}
{"grpc.code":"OK","grpc.method":"CreateFork","grpc.request.deadline":"2019-11-06T02:32:06Z","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019
-11-05T20:30:26Z","grpc.time_ms":61.691,"level":"info","msg":"finished streaming call with code OK","pid":13825,"span.kind":"server","system":"grpc","time":"
2019-11-05T20:30:26Z"}
Edited by Will Chandler (ex-GitLab)