CI: backup/restore testing leaves application in mixed state
Summary
As found 2020-02-03, part of the CI testing is leaving the application in a mixed state, which can cause issues with usability and breaks QA operation.
Effectively:
- Database is in a near-pristine state after the restore
- Gitaly node has data on disk (
/home/git/repositories/@hashed
) which prevents use ability to create projects under/root/xyz
Steps to reproduce
- Previous Helm deployment in a SUCCESS state, so no "purge" will occur.
- The
production_specs_gke
Job will perform a restore from a known-state backup, using backups_spec.rb- The database will have 1 project
-
gitlab-rails runner 'Project.last!
will render1
-
- As QA ran prior, contents will be present on Gitaly's disk.
-
find /home/git/repositories/\@hashed -mindepth 2 -maxdepth 2 -type d | wc -l
will render many hundreds, probably > 1000.
-
- The database will have 1 project
- Login as
root
into the "production" instance, and attempt to create a new project (blank, from template, import from URL)
Configuration used
Standard
Current behavior
Gitaly logs
gke-production-a4b9oa-gitaly-0 gitaly {"correlation_id":"01EXM49NHRZZAK88ARNKRACTYK","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.method":"RepositoryExists","grpc.request.deadline":"2021-02-03T14:21:28Z","grpc.request.fullMethod":"/gitaly.RepositoryService/RepositoryExists","grpc.request.glProjectPath":"","grpc.request.glRepository":"","grpc.request.repoPath":"@hashed/ef/2d/ef2d127de37b942baad06145e54b0c619a1f22327b2ebbcfbec78f5564afe39d.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"@hashed","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2021-02-03T14:21:18Z","grpc.time_ms":0.293,"level":"info","msg":"finished unary call with code OK","peer.address":"10.40.12.145:56382","pid":9,"remote_ip":"174.54.214.92","span.kind":"server","system":"grpc","time":"2021-02-03T14:21:18.476Z","username":"root"}
gke-production-a4b9oa-gitaly-0 gitaly {"correlation_id":"01EXM49NHRZZAK88ARNKRACTYK","error":"rpc error: code = NotFound desc = GetRepoPath: not a git repository: \"/home/git/repositories/root/asdfasdfasdfasdf.git\"","grpc.code":"NotFound","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.method":"FindDefaultBranchName","grpc.request.deadline":"2021-02-03T14:21:28Z","grpc.request.fullMethod":"/gitaly.RefService/FindDefaultBranchName","grpc.request.glProjectPath":"root/asdfasdfasdfasdf","grpc.request.glRepository":"project-","grpc.request.repoPath":"root/asdfasdfasdfasdf.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"root","grpc.service":"gitaly.RefService","grpc.start_time":"2021-02-03T14:21:18Z","grpc.time_ms":0.35,"level":"info","msg":"finished unary call with code NotFound","peer.address":"10.40.12.145:56382","pid":9,"remote_ip":"174.54.214.92","span.kind":"server","system":"grpc","time":"2021-02-03T14:21:18.873Z","username":"root"}
gke-production-a4b9oa-gitaly-0 gitaly {"correlation_id":"01EXM49NHRZZAK88ARNKRACTYK","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.method":"RepositoryExists","grpc.request.deadline":"2021-02-03T14:21:28Z","grpc.request.fullMethod":"/gitaly.RepositoryService/RepositoryExists","grpc.request.glProjectPath":"root/asdfasdfasdfasdf","grpc.request.glRepository":"project-","grpc.request.repoPath":"root/asdfasdfasdfasdf.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"root","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2021-02-03T14:21:18Z","grpc.time_ms":0.16,"level":"info","msg":"finished unary call with code OK","peer.address":"10.40.12.145:56382","pid":9,"remote_ip":"174.54.214.92","span.kind":"server","system":"grpc","time":"2021-02-03T14:21:18.878Z","username":"root"}
Webservice logs
gke-production-a4b9oa-webservice-default-7889cb7784-x8z57 webservice *** /var/log/gitlab/application_json.log ***
gke-production-a4b9oa-webservice-default-7889cb7784-x8z57 webservice {"severity":"ERROR","time":"2021-02-03T14:21:18.481Z","correlation_id":"01EXM49NHRZZAK88ARNKRACTYK","message":"Unable to save project. Error: uncaught throw :abort"}
Expected behavior
Success
Versions
- Chart: 59c6a95e
- Platform:
- Cloud: All
Edited by Jason Plum