Can't push using Deploy Key after update from 12.3.x to 12.4.1 (This action cannot be performed by internal users, Your account has been blocked)
Problem to solve
Some customers automation scripts are broken because in some cases "git push" using deploy keys is not working any more. This happened, for examples, for deploy keys that were created by users, whose accounts are blocked now or keys having null user_id in the database.
Intended users
Further details
Use case
I created 4 ssh keys, added them as global from admin interface. In the database table "keys" they had "user_id" set to my user's id. I left first as is, for others set "user_id" field to: NULL, id of blocked user, id of another user accordingly. When I ssh to the gitlab instance with this keys I get greetings like this:
- key1: Welcome to GitLab, @[my_user]
- key2: Welcome to GitLab, Anonymous!
- key3: Welcome to GitLab, @[blocked_user]
- key4: Welcome to GitLab, @[other_user]
I find it strange also, that deploy key is linked in such a way with its creator, but this is offtopic here.
Then I prepared a private project: created an empty project, enabled those depoy keys in it, set write permission, cloned, made initial commit, pushed. After that tested pull/push operations with different deploy keys.
Clone or pull works with all those keys.
Push operation works only with the first key, for other keys I receive such errors:
- key2 (null user)
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 256 bytes | 256.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: GitLab: This action cannot be performed by internal users
To ssh://gitlab/user/test-keys.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@gitlab/user/test-keys.git'
- key3 (blocked user)
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 256 bytes | 256.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: GitLab: Your account has been blocked.
To ssh://gitlab/user/test-keys.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@gitlab/user/test-keys.git'
- key4 (other user):
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 256 bytes | 256.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: GitLab: The project you were looking for could not be found.
To ssh://gitlab/user/test-keys.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@gitlab/user/test-keys.git'
When I was dealing with one of colleagues issues, I was able to push with key "beloning" to other user, who had Developer permission for the repository. Do not know what type of access other user needs in this case, have not tested all the cases.
Proposed workaround that works for some users:
When I noticed that file don't add keys, I delete file /var/opt/gitlab/.ssh/authorized_keys and stared command sudo gitlab-ctl reconfigure.
This file create. Via web-interface I delete all keys, and add their again.
It worked!
Proposal
As per #35779 (comment 250710439) there is a workaround offered which basically queries for expired users and replaces the user with a valid user while maintaining the deploy key - so that in fact no project configuration needs to change since the deploy key stays the same. Under the admin area->deploy keys currently allows you to create instance-wide SSH deploy keys. Can we add a list there of the anomalies - meaning display a table only for blocked and expired users
Project name | Project ID | User status | User ID | Deploy Key |
---|
The admin can edit the user ID from this console and this will make the change in the DB without needed to run anything.
Also we would need to capture this in an audit log
Permissions and Security
There are scenarios, where the blocked user still has/owns the private SSH key which pairs with the public SSH key imported into the project. Would not that raise a security concern?
Documentation
Availability & Testing
What does success look like, and how can we measure that?
What is the type of buyer?
Links / references
/label feature
Old issue for safe keeping
After updating from version 12.3.x to 12.4.1, I'm getting complaints from colleagues that they have their automation scripts broken because in some cases "git push" using deploy keys is not working any more. This happened, for examples, for deploy keys that were created by users, whose accounts are blocked now or keys having null user_id in the database.
I have also find similar issue here: https://forum.gitlab.com/t/unable-to-push-to-repo-after-updating-gitlab/31088
Steps to reproduce
I created 4 ssh keys, added them as global from admin interface. In the database table "keys" they had "user_id" set to my user's id. I left first as is, for others set "user_id" field to: NULL, id of blocked user, id of another user accordingly. When I ssh to the gitlab instance with this keys I get greetings like this:
- key1: Welcome to GitLab, @[my_user]
- key2: Welcome to GitLab, Anonymous!
- key3: Welcome to GitLab, @[blocked_user]
- key4: Welcome to GitLab, @[other_user]
I find it strange also, that deploy key is linked in such a way with its creator, but this is offtopic here.
Then I prepared a private project: created an empty project, enabled those depoy keys in it, set write permission, cloned, made inital commit, pushed. After that tested pull/push operations with different deploy keys.
Clone or pull works with all those keys.
Push operation works only with the first key, for other keys I receive such errors:
- key2 (null user)
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 256 bytes | 256.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: GitLab: This action cannot be performed by internal users
To ssh://gitlab/user/test-keys.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@gitlab/user/test-keys.git'
- key3 (blocked user)
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 256 bytes | 256.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: GitLab: Your account has been blocked.
To ssh://gitlab/user/test-keys.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@gitlab/user/test-keys.git'
- key4 (other user):
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 256 bytes | 256.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: GitLab: The project you were looking for could not be found.
To ssh://gitlab/user/test-keys.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@gitlab/user/test-keys.git'
When I was dealing with one of colleagues issues, I was able to push with key "beloning" to other user, who had Developer permission for the repository. Do not know what type of access other user needs in this case, have not tested all the cases.
What is the current bug behavior?
Deploy keys that were working on the 12.3.x version, are no longer working and I see no reason why user, that created them should affect its behaviour. Because now "anonymous" keys are not working, as keys, created by some categories of users. And I see no work case how to make a deploy key, that will work independently of the status of some account.
What is the expected correct behavior?
It should be possible to use deploy key with write access, that was added by blocked or some unrelated user. A working script should continue working if the deploy key is still enabled in the repo, no matter that some user was blocked. Because currently somebody have to add the key for the script to work, but later it can stop working on the project and that should not break it.
Proposal
Create a way that will replace an existing deploy key with a new one in the UI with the correct permissions (will in fact replace the entry in the DB), that will in turn change the keys for the existing projects