Deleting project deletes deploy keys?
Summary
I deleted a project that was not in use but which had some deploy keys attached to it. After I did so the deploy keys were deleted from all projects that had those keys (about 26). I re-added all of the deleted deploy keys and now the deploy keys are added back but they cannot clone any repositories!
$ git clone git@gitlab.example.com:my/repo.git
Cloning into 'repo'...
GitLab: The project you were looking for could not be found.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
However I can authenticate:
$ ssh -T git@gitlab.example.com
Welcome to GitLab, Anonymous!
The error logs show access denied:
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
I, [2017-08-28T20:57:05.056008 #32071] INFO -- : POST http://127.0.0.1:8080/api/v4/internal/allowed 0.01895
W, [2017-08-28T20:57:05.056340 #32071] WARN -- : gitlab-shell: Access denied for git command <git-upload-pack 'my/repo.git'> by user with key key-19.
But there is no longer a "key-19" in the database:
# docker exec -it gitlab gitlab-psql gitlabhq_production
psql (9.6.3)
Type "help" for help.
gitlabhq_production=# select id, user_id, title from keys where type LiKE 'DeployKey';
id | user_id | title
----+---------+--------------------------
11 | 10 | ci
20 | 1 | mgr2
23 | 1 | colin@app1
16 | 1 | mgr1
24 | 1 | admin1.sh
25 | 1 | www-data@dev
22 | 1 | staging1
(7 rows)
The keys that didn't exist before the old keys were deleted work, it is the keys that existed before but got deleted and re-added that are broken.
Steps to reproduce
- Create two projects with the same Deploy Key.
- Delete one of the projects.
What is the current bug behavior?
The deploy keys get deleted from the other projects they were associated with. After re-adding the deploy keys they are no longer working for cloning projects (although authentication works).
What is the expected correct behavior?
Deleting one project should have absolutely no effect on another project.
Relevant logs and/or screenshots
See above for logs.
Manual steps taken to repair
I deleted the rows from deploy_keys_projects
which referenced keys that no longer existed:
gitlabhq_production=# delete from deploy_keys_projects where deploy_key_id IN(17,19,21);
DELETE 78
I rebuilt the authorized_keys
file and cleared cache and repaired hooks:
root@gitlab:/# gitlab-rake gitlab:shell:setup
This will rebuild an authorized_keys file.
You will lose any data stored in authorized_keys file.
Do you want to continue (yes/no)? yes
....................
root@gitlab:/opt/gitlab# gitlab-rake cache:clear
root@gitlab:/opt/gitlab# gitlab-rake gitlab:shell:create_hooks
Creating/Repairing hooks symlinks for all repositories
done
However, the problem persists. Please help me get this error fixed!!
I restarted the omnibus container and now it works. I think the restart fixed it because it killed my persistent connection which I had forgotten about and was probably preventing the new keys from being picked up from my test location.
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 10.5.0 Redis Version: 3.2.5 Git Version: 2.13.0 Sidekiq Version:5.0.0 Go Version: unknown
GitLab information Version: 9.4.1-ee Revision: 1ae0012 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.3 URL: https://gitlab.bss-llc.com HTTP Clone URL: https://gitlab.bss-llc.com/some-group/some-project.git SSH Clone URL: git@gitlab.bss-llc.com:some-group/some-project.git Elasticsearch: yes Geo: no Using LDAP: no Using Omniauth: no
GitLab Shell Version: 5.3.1 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 5.3.1 ? ... OK (5.3.1) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 1/1 ... ok 3/2 ... ok 2/4 ... ok 2/5 ... ok 2/15 ... ok 2/16 ... ok 2/17 ... ok 2/18 ... ok 2/20 ... ok 2/21 ... ok 2/22 ... ok 2/23 ... ok 2/25 ... ok 2/26 ... ok 2/27 ... ok 2/28 ... ok 11/29 ... ok 17/30 ... ok 2/31 ... ok 17/32 ... ok 17/33 ... ok 17/34 ... ok 17/35 ... ok 17/36 ... ok 17/38 ... ok 17/40 ... ok 2/42 ... ok 17/43 ... ok 17/44 ... ok 1/45 ... ok 14/46 ... ok 14/47 ... ok 14/48 ... ok 14/49 ... ok 14/50 ... ok 14/51 ... ok 2/52 ... repository is empty 2/54 ... repository is empty 11/55 ... repository is empty 2/57 ... repository is empty 2/58 ... ok 17/59 ... ok 2/60 ... ok 2/61 ... ok 2/62 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1/1 ... yes 3/2 ... yes 2/4 ... yes 2/5 ... yes 2/15 ... yes 2/16 ... yes 2/17 ... yes 2/18 ... yes 2/20 ... yes 2/21 ... yes 2/22 ... yes 2/23 ... yes 2/25 ... yes 2/26 ... yes 2/27 ... yes 2/28 ... yes 11/29 ... yes 17/30 ... yes 2/31 ... yes 17/32 ... yes 17/33 ... yes 17/34 ... yes 17/35 ... yes 17/36 ... yes 17/38 ... yes 17/40 ... yes 2/42 ... yes 17/43 ... yes 17/44 ... yes 1/45 ... yes 14/46 ... yes 14/47 ... yes 14/48 ... yes 14/49 ... yes 14/50 ... yes 14/51 ... yes 2/52 ... yes 2/54 ... yes 11/55 ... yes 2/57 ... yes 2/58 ... yes 17/59 ... yes 2/60 ... yes 2/61 ... yes 2/62 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.3.3) Git version >= 2.7.3 ? ... yes (2.13.0) Active users: ... 24 Elasticsearch version 5.1 - 5.3? ... yes (5.3.2)
Checking GitLab ... Finished
Possible fixes
Deleting project should only delete rows in deploy_keys_projects
and not keys
. Also a foreign key with ON DELETE RESTRICT
to prevent breaking data integrity would be good.