Feature Request: Allow runners to push via their CI token

@l.stoetzel I see the need for that. We introduced the Release feature. Maybe it would be better just to allow that?

This would make the Releases to work pretty much the same:

Create or update a release with version and description
Attach files to release

The release created that way would not trigger the builds (it's not needed).

What do you think?

I would love to see this feature. I've actually just spent the last hour or so trying to figure out how to do it... I guess I should have Googled first .

My usage case is: I have a repo that contains various formats of the same data (YAML, JSON, CSV, XML, etc.). I have a script that take the master YAML file and converts it to the others. Basically every time I push changes I forget to run that conversion script, so I figured I'd try and get CI to run it for me.

As for infinite loops, one potential solution would be to force-append [ci skip] to the commit message.

@dougthor42: In my envorinment the solution to this was:

Generate a new SSH key
Create a new GitLab account for your runner & add the key to this account
Add the key to the system account the runner runs under

My process to push the changes is:

# for your information
whoami
printenv

# we need to extract the ssh/git URL as the runner uses a tokenized URL
export CI_PUSH_REPO=`echo $CI_BUILD_REPO | perl -pe 's#.*@(.+?(\:\d+)?)/#git@\1:#'`

# runner runs on a detached HEAD, create a temporary local branch for editing
git checkout -b ci_processing
git config --global user.name "My Runner"
git config --global user.email "runner@gitlab.example.org"
git remote set-url --push origin "${CI_PUSH_REPO}"

# make your changes
touch test.txt

# push changes
# always return true so that the build does not fail if there are no changes
git push origin ci_processing:${CI_BUILD_REF_NAME} || true

Thanks a lot @l.stoetzel. I had the same "create a runner user account" idea, but I was missing some of the details such as extracting the ssh/git URL and the detached HEAD state. That helps a bunch!

I also realized a few minutes ago that I could also use a git pre-commit or pre-push hook to run the conversion script. It doesn't resolve the proposed GitLab idea (which I'm still in favor of), but it does mean I don't forget to convert my files!

@dougthor42

You can push back with this:

git push http://gitlab.com/group/project.git HEAD:my-branch

I would like to see that at some point, but this is not easy to implement currently. We will think about it. Thanks

I would probably tend to agree with proposal A, however I would suggest that it be configurable if it is limited to only the branch that was pushed to. Personally, I would like a workflow where developer can tag a commit with some 'staging' tag, and then in my .gitlab-ci.yml I have tasks that run only when tagged with 'staging' that will do all the required tests and actions to bump versions and promote develop to master and push back new versions to both develop and master.

An example from a gitlab-ci.yml using maven and jgitflow using the workaround suggested by @l.stoetzel to demonstrate desired functionality:

release:
  stage: release
  script:
    - '[ "$(git rev-parse HEAD)" == "$(git rev-parse develop)" ]'
    - git remote set-url --push origin $(echo $CI_BUILD_REPO | perl -pe 's#.*@(.+?(\:\d+)?)/#git@\1:#')
    - git checkout master
    - git checkout develop
    - mvn -B jgitflow:release-start
    - mvn -B jgitflow:release-finish
    - git push --all
  only:
    - staging
  tags:
    - maven

Although these work arounds work, they do pose security risks as environment variables or files within the container can be displayed in the build log if the .gitlab-ci.yml is modified to do so, however it would seem that the OAuth secret is always masked.

I would really like this feature so that I can easily compile my pages assets to another branch (a.k.a gl-pages). Compiling to another branch allows me see the contents of the build without downloading an archive every time I want to look, and serves as example output for my app.

Moved from gitlab-org/gitlab-ci#372

Added ~19173 label

Milestone changed to %9.4 in gitlab-runner

This is a feature I would like as well. There are lots of work arounds, but proposal A is nice - allowing a particular runner using the token to push only to that repo it has access to. I personally use it a lot for tagging releases.

@ayufan Proposal B seems to fit in well with our current thoughts on permissions.

Current thoughts on permissions are in https://gitlab.com/gitlab-org/gitlab-ce/issues/18994

Nvidia is also interested in this https://news.ycombinator.com/item?id=12490487

Should this be marked for 8.12 as the milestone as well?

gitlab-ce#18994 is on track for shipping in 8.12, but only includes read access. Write access is not planned for 8.12, although it's a likely extension. Because of the security challenges, we want to be a little cautious here and do one step at a time. gitlab-ce#18994 feels safe, and is the right direction. If all goes well, it will evolve.

Added ~14213 label

If this Issue will also address https://gitlab.com/gitlab-org/gitlab-ci/issues/105 then this customer is very interested: https://gitlab.my.salesforce.com/00161000005dEuv

I will agree that pushing tags from the CI as mentioned in https://gitlab.com/gitlab-org/gitlab-ci/issues/105 is very important. Otherwise, the workaround with SSH keys is not so secure as SSH keys are visible as Variables (or another workaround involving runner image having the key from its ENV in the beginning is needed). Lot of workarounds for pushing tags.

@stormbringerdp

I think that allowing to push tags is very viable option. We should consider adding that on top of current changes :)

I see a big benefit of having that myself :)

FWIW, my scenario for this is to do a merge from development into master as part of my Manual Promotion step in my Pipeline. so...

promote_to_production:
  <<: *only_dev
  stage: promote
  script: git push origin development:master
  when: manual

right now, i get this

$ git push origin development:master
fatal: Authentication failed for 'https://gitlab-ci-token:xxxxxx@gitlab.imshealth.com/channels/TheChunnel.git/'
ERROR: Build failed: exit code 1

but i'd love to be able to just click Promote to make this happen.

@Kevin.OBrien I'm currently using a special project user with it's own permission, for commits within pipelines..

Any updates on this? Proposal B sounds good to me.

This customer spent several days trying to debug this issue: https://gitlab.my.salesforce.com/00361000016jrse Is there any way for us to provide an error message that might help them understand the issue quicker until we can provide a remedy?

I have the (almost) exact usecase as @andycunn; we want a button to merge to a testing stage, preferably without having to mess around with ssh keys; proposal B seems fine for that purpose (and in line with the current config) isn't this quite a small thing to add? (as we have gitlab-ce#18994); maybe for security make it an opt-in in /admin or per-group or something?

p.s.: the current workarounds prevent us from completing our chain (we lose the identity of the dev when the second pipeline is triggered)

I like proposal B. Having traceability to a user is a lot better than having non personal accounts write to the repository.

I'm pretty sure either would imply traceability, the difference being the actual rights; however I do think that proposal B is more in line with gitlab's way of thinking

Well, it would seem that with gitlab-ce#18994, the approach being taken is closer to Proposal B and given that we already have this functionality where builds use the permissions of the person who pushed I think this should be kept. One of the reasons I was more of an advocate for Proposal A was that I would like to have different permissions for the user and the CI build (i.e. for the release workflow, I don't want to be allowed to push to master from my own machine but if I trigger a release pipeline, I want that pipeline to be able to push to master).

Should this continue, I would like to see an additional option in the protected branch settings which would allow for this protection either through additional roles ("CI Runner") or through an option (checkbox).

I would probably go for the checkbox option (after the Allowed to push: option) similar to Only allow pushes from CI runners such that pushes will be allowed from CI jobs where the owner of the pipeline is listed in the Allowed to push list.

The one limitation of this implementation is that it affect all users, roles and groups and would not allow for the scenario where, for example, a Master would be able to push regardless but developers must push using a CI pipeline. I would think that this scenario is probably a more limited case and in the case where this is true, the weaker permissions can be settled on for now (rather than complicating the UI with an option per role/user/group which is granted permission).

Restrictions on which pipelines are given push permissions may also be required, possibly achieved with a regular expression to match the name of the pipelines which are granted push permissions in the CI token (in my use-case, this would be the release branch).

Also it can be done using

git remote set-url --push origin "git@gitlab.com:${CI_PROJECT_PATH}.git"

and passing ~/.ssh as volume into container. If host has write access to repo, then container will has too.

PS For host other than gitlab.com you can hardcode it (I think that you don't often change host for your runner) or get it from env variable + regex

How do you avoid the loop with proposal A or proposal B? I recently set up a project to push git tags for a prereleases ... but I also pushed the updated version file and started a loop. Since another job calls the GitLab API I wanted to use a token rather than ssh keys.

variables:
  CI_REPOSITORY_URL: https://$GIT_ACCESS_USER:$GIT_ACCESS_TOKEN@gitlab.com/$CI_PROJECT_PATH.git 
script:
  - git config --global user.name $GIT_ACCESS_USER
  - git config --global user.email $GIT_ACCESS_EMAIL
  - npm run version -- --prerelease $CI_COMMIT_REF_SLUG
  - git push --follow-tags $CI_REPOSITORY_URL HEAD:$CI_COMMIT_REF_NAME

The Problems:

The push is not tied to the user that triggered the pipeline
If the push updates a file, it triggers another branch based pipeline
Hassle setting up the GIT_ACCESS_TOKEN
The personal access token is long lived and this seems to significantly increase my vulnerability footprint

Proposed Usage:

What about an option in the pipeline settings to select which (regex on job names?) jobs can have write access for the existing CI_JOB_TOKEN?
This would not introduce a new set of variables and would allow users to limit the scope of which jobs (and thus branches/tags) can have the write perms.
This would follow gitlab-ce#18994 but would add the additional exception of the write perms to include the identified jobs and writing to ones own container registry.

After a trigger...

curl -X POST -F "variables[release]=true" -F token=thetoken -F ref=master http://gitlab.local/api/v4/projects/3/trigger/pipeline

Note the release environment variable

My release script was going to do this...

# release failure exit code
f_release=3

echo "Releasing: $release";
if [ "$release" == "true" ]; then
  git config user.email "$GITLAB_USER_EMAIL"
  git config user.name "Runner = $CI_RUNNER_DESCRIPTION"
  git config push.default matching
  npm version patch -m "Release %s [ci skip]" || exit $f_release;
  git push || exit $f_release;
  git push --tags || exit $f_release;
fi

The result would have been that the pipelines don't run until the tag occurs, which is exactly when I want the standard pipeline, including deployment, to occur.

But the result was...

fatal: Authentication failed for 'http://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.local/trent/project.git/'

So yes, just leaving the auth tokens open until the job is done, would make this work, wouldn't it?

My yaml I'm hacking together right now, learning gitlab CD, is attached. .gitlab-ci.yml

It seems to me that there are two aspects to this. One is the granularity of access, e.g. permission to push to master, push only to current branch, push with user's privileges etc. The other is the mechanism by which the push request is secured between the runner and the repo. Currently, this is secured with an asymmetric key pair that the user has to set up and configure, i.e the Deploy Key. I am going to argue that gitlab-ci should set up the keys and a per session/runner basis because it is simpler for the users and more secure.

Currently, to use a Deploy Key, I have to generate a key pair; put the private part in a secret variable, and set the public part as a Deploy Key. Then I have to use the secret variable with the private key to configure ssh on the runner. From a user perspective, this seem like I am setting up the application to authenticate itself to itself, and it is extra work that the user have to learn and get right. Furthermore, it is using a permanent secret for the runner to authenticate itself - as other have mentioned, this can be leaked through log files etc.

I can see no benefit from making users go through all this manual key set up - gitlab could do it itself if the user clicks on the "allow push" checkbox. All that is really needed is a secret session key, which can be a simpler symmetric key. This could be automatically generated and passed to the runner much like the secret variable with the private key is passed currently. The security would be improved because the runner now only has a session key which should be invalidated when the runner finishes the job.

There are various ways this could be implemented, and I am guessing that a way that fits with the existing repo security will be preferable. Therefore, a symmetric shared secret key could be just a long random (session) password. Or if a public/private asymmetric key pair is used then it could be automatically set up in the same way as the current mechanism. (Generating an asymmetric key pair takes longer though, but it would not have to be as long as a permanent asymmetric key pair so a shorter length may offset that a bit.)

The significant improvement is that the users just check a checkbox rather than manage the key set up themselves.

As an addendum to @simon.ward's comment, a very closely related issue to this is installing subpackages on build via something like composer. Right now, the only solution is adding a private ssh key for a user who has access to all required packages for the build to take place. This is easy to manage for individual servers using the deploy keys api and a list of required packages but quite difficult to manage for automated testing where deploy keys have a lifespan of 5 minutes.

Creating an auto-generated key pair would make this much easier not just for Push but also for Pull. I understand that it's out of scope of this specific issue, however, the same solution could be used to solve two different problems. I'd assume package manager support is actually a much more common problem.

I think there are going to be many use cases and workflows. The gitlab-ci-token was configured to allow for multi-projects when accessing code. So in that case there could be cases where it now also requires the ability to push back to those.

For example I work on a repository that controls keeping things in sync across many other repositories. So it clones down a YAML list of repo's see's if it needs to update anything and then has the ability to "update" those managed repos. (I've added some additional tooling around then have build step that will open a MR for that pushed regex matching branch).

I don't see that as being something that wouldn't be a necessity for others as well and would avoid having to run these steps outside of a CI pipeline.

I like the idea of enabling/disabling allowing pushes from runner as maybe some projects don't want to allow that ability and even going a step further if there was a way to determine which project the runner was run from and give permissions to only a runner that was run via that project.

So for example if I have config_sync repo and it checks repos project1,2,3 you could enable project1,2,3 to only allow push from ci-runner if repo was config_sync... this would disable ANY ci-runner from having push access and restrict down to where the runner came from (probably with one of the other CI_ environment variables).

In case it is of use to anybody as an interim solution, I am doing something along the lines of what is described here https://nicolaw.uk/GitLabCiPushingWrites

I store the private part of the SSH deploy key as a secret variable, which I then write to disk temporarily during the CI job pipeline (and remove after of course).

It works pretty well for me so far. (Thanks and credit to the hints from other comments posted above).

@nicolaw I think especially in a Docker container the official way with ssh-agent has several advantages, like not storing the key on disk and having no need to override GIT_SSH_COMMAND.

It shares the disadvantage that SSH are difficult to work with because they contain newlines.

Please note, that after 9.X version, some CI_BUILD_XX variables have been renamed: https://docs.gitlab.com/ee/ci/variables/#9-0-renaming

This broke all my CI builds, which is awesome!

My use case is having a file that contains both data obtained automatically and manual edits. I've set it up so that the manual and automatic edits don't conflict, but having a dedicated user for pushes is quite a bit of micromanagement.

mentioned in issue awakened-heart-sangha/main-site#40

This broke all my CI builds, which is awesome!

@lunemec Thats not true, the old ones are still exposed.

Is there a path forward for this?

@bikebilly What is our current standpoint on this? Another customer asking: https://gitlab.zendesk.com/agent/tickets/89715

I am not sure of the current status of this ticket or if I have hit this. When I try and push back to the repo I checked out in a CI build I get the following error:

error: src refspec experimental does not match any.
error: failed to push some refs to 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.com/ganges/grpc-ruby-code.git'

Full log (private project) available at https://gitlab.com/ganges/grpc-ruby-code/-/jobs/57572725

The reason I am not sure if I have hit this issue or not is because I have had issues with really vague / misleading error messaging from gitlab around permissions (e.g. returning not found messaging instead of access denied) and I cannot see anyone else posting the actual error that appears for them. So:

Am I hitting the issue described here?
What is the current status on implementing it?

The desribed option B in this issue seems to be the way Gitlab is heading. (e.g. being able to use the token to push an image to the docker container registry or using it to trigger a pipeline from another pipeline in the dev's name)

Incase you want to push changes to a protected branch without fetching all of the repo you can follow the following gist https://gist.github.com/regevbr/cf6d82dba1691123783ccfa22a304f9b which leverages the api to create a branch and then merge it using a pull reques

Proposal B sounds great but it is sad that this feature proposal has ~130 thumbs up, several customer requests and still no update...

gitlab-ce#41084 is an proposal to get general more access from within a CI-job with a job-token.

This would include:

the registry (push/pull/delete from and to private/public projects)
Push/Pull to repository (e.g. automatic updates of change logs for automated releases....etc)
API-access to
- retrieve Release info,
- trigger new pipeline,
- issue a deployment token... etc.

Above for all projects where the user who triggered the pipeline has access to!

This proposal would be the ground work to generate an extend token that can be made only available on a safe runner.

And it seems to need an extended job token seen the security issues that would arise from adding more permissions to the current CI-JOB-TOKEN,

whereas the CI-JOB-TOKEN is always made available as an environment variable, also in unsafe executers like shell,ssh and privileged docker containers, allowing malicious users to obtain the token.

And this security concern is probably the main reason why this much needed issue is lying around 2 years with no action!

Any support is welcome at gitlab-ce#41084 !

mentioned in commit monasca-installer/monasca-installer@479f684b

+1 on this feature

Having to use deploy keys and ssh is very tedious, error prone, and insecure

It would also be nice if somehow gitlab would automatically know or let us pass a param on push that would stop it from double triggering jobs

Using excet: tags seems like a hammer since tags might be pushed for various reasons and not just by a specific CI stage

This is also of importance to customer - https://gitlab.my.salesforce.com/0066100000JKO3e.

It is of importance to this customer too

Gitlab needs a good module system so that we can implement these features ourselves.

Hi, I want to know the status of this ticket. We have been ssh for a while, but we are looking forward to see the token being valid for push also.

@aliksend can you please share some details about your volumes and ci configuration to end up with a working git push?

@d-kopriwa I tried to do it about year ago. For now I don't need it but may be return to this question in future.

Added a but wanted to say how I stopped the infinite loop also.

I use a docker executor and the host machine has the SSH keys so I just add a volume to my config.toml:

volumes = ["/cache", "/home/ec2-user/.ssh:/root/.ssh"]

And of course the public key is added to a user, I'll call the user my-bot for this. So now I should be able to push any changes like the commit and tag npm version produces. Good!

But if I were to push the commit and tag it will then run another pipeline run and I'm in an infinite loop. Since I'm using that "bot" user, I now can check this variable within an except block. Here is an example:

dev build:
  stage: build
  except:
    variables:
      - $GITLAB_USER_LOGIN == "my-bot"
  only:
    - develop@username/reponame
  before_script:
    - npm version patch
  script:
    - echo "Do build here before pushing the version bump in case the build fails"
  after_script:
    - git push git@my-repo-url:$CI_PROJECT_PATH.git HEAD:$CI_COMMIT_REF_NAME --follow-tags

I chose to use $GITLAB_USER_LOGIN instead of $GITLAB_USER_ID because the username is more descriptive and someone else looking at this would understand it better. You could use a pattern to match multiple usernames too if that's your thing. The except is also something I throw into an anchored job and merge it in as I have multiple build jobs for different environments.

As said above, there are multiple ways to change the URL you push to. I chose to inline it within the git push command and even use variables to create things a bit more. Couldn't find a variable that gave me my install page like gitlab.foo.com, only found some with protocol and everything and I didn't want to parse it so it's hardcoded which is fine, don't think that will change.

The only is there to disable the build job(s) in a fork.

@mitchellsimoens The problem with your approach is that the docker container can now reach all of your SSH keys.

My approach was to temporarily add ssh keys to the key store and then blow up the ~/.ssh directory after the CI job completes.

Sure... except it only has one ssh key that we actually rotate weekly (it's a build agent machine so the ssh key isn't used in lots of places so easily to swap out). I didn't like people suggesting to add a private variable with $SSH_PRIVATE_KEY, that always seems like a bad idea to me.

How is that any worse or different than what you're doing? If the key is exposed to the CI job then you have to trust that people who can run jobs where it is exposed aren't going to abuse that.

It's not just about trust (our Gitlab is behind a VPN anyway). It's also about managing our rotated keys. Adding a private variable also adds steps to replace it for every CI build that would define that variable. With what I'm doing, I just rotate the keys on the box, update the public key in my gitlab "bot" user and I'm done. Add a new project? Sure, already supported! Also, this process is scripted so it happens via a cron job and while it can be manually triggered, there simply isn't anything anyone has to do. So instead of modifying the script to create or update project variables and create a dependency to update that when a new project is added (that will always be a high risk of being missed especially since at any point the people that know about this setup may leave the project), I just have two things... create a new ssh key pair, update public key on user.

Our CI jobs are not supposed to reach all projects so we go with the $SSH_PRIVATE_KEY approach. Each project has its own deploy keys.

We don't want each and every project (including newly created) to be accessable by all containers. That sounds like a bad idea to me.

We don't have cronjobs or other things that we have to maintain.

Simply, each project that requires push via CI has its own deploy key, has its own $SSH_PRIVATE_KEY variable and that is all.

Great, you architected your projects to suit your needs and I'm architecting mine to suit my needs. This is how the world works right? Not all projects are the same . So far, all of my projects need the ssh keys to push to git as they all have that npm version command so I need to push that back up to the git repo. So since all of them need keys, client wants those keys rotated weekly, the need to automate the process via something like a cronjob is needed... for me, maybe not for you. Great. I'm glad you have success!

@mitchellsimoens you also are using docker executor I'm assuming which is great but if you use something like kubernetes to run your executors, unless you bake the SSH keys on every worker node and potential scaled worker node, this method wouldn't scale and would have security implications as previously mentioned.

Gotta love pointing out the except stopping the infinite loop (literally the first sentence in my orig reply) and having people jumping on me for how I'm adding .ssh keys. For me, it works, it works very well and scales for me without any security issues other solution (you know, the $SSH_PRIVATE_KEY solution) would also have. Ok, for something I'm not using the answer would be different but that's for something I'm not using which is pretty obvious from my post.

So thank you for pointing out that my solution would not work for kubernetes as that will no doubt help people using that to give a more clear overall picture and reaffirming that what this feature request is about really needs to be implemented instead of everyone working around it like we are.

@mitchellsimoens everyone has their own solutions... it was merely a matter of raising concerns since there are times people read through other peoples "solutions" and blindly follow... what works for you or me might not work for everyone and also might be "secure" enough for us but definitely not for public consumption. So if anything it was more of a PSA.

I tried to read across this and a few related tickets, but it's a lot of information, so... To quickly summarize this discussion:

There is no official and secure way of pushing back changes to the repository within a gitlab ci workflow, for instance after updating the version of a project.
There are several workarounds which involve adding ssh keys to runners or fiddle with deploy tokens.
There is no known roadmap for implementing an easy way of pushing back changes to the repo.

Right?

@athaphian that sums it about up.

Anyhow please also help to upvote gitlab-ce#41084 a request to get an extended job token.

@All we have self-hosted gitlab on our server. I am not allowed to create a new account for a runner because that entails making a real email on our server and doing many related task to it. I was trying to push from runner using a key pair, by adding public key to the repo's deploy key and using the private key(using a secret variable) in runner job and creating id_rsa file with every possible permission like 600, 700, 644. But it keeps on giving me either permission are too open or public key denied access.

You can checkout my gitlab-ci yml configuration here https://gitlab.com/snippets/1733511

@raghavgarg1257 this error as nothing to do with GitLab, you can google it.

Also, it is possible to push with a personal access token, you don't need a rsa key.

@d-kopriwa Thanks for the reply. I have manually tried the keys on the server and they are working fine if try to push.

about personal access token, can you please enlighten me more? I have used it to push using rest api call but was unsure how to use in the repo, like should I add it in remote or give the whole URL while pushing. Can you tell me how should I do it.

Also, I am wondering if my hosted gitlab have some problem because of the port. git remote set-url origin ssh://git@hostname.com:33022/path/to/repo.git could there be any problem because of it?

@raghavgarg1257

Can you tell me how should I do it.

You can try to do something like this within a bash script executed during before_script:

if [[ -v "GL_TOKEN" || -v "GITLAB_TOKEN" ]]; then
  if [[ "${CI_PROJECT_URL}" =~ (([^/]*/){3}) ]]; then
    mkdir -p $HOME/.config/git
    echo "${BASH_REMATCH[1]/:\/\//://gitlab-ci-token:${GL_TOKEN:-$GITLAB_TOKEN}@}" > $HOME/.config/git/credentials
    git config --global credential.helper store
  fi
fi

Also, I am wondering if my hosted gitlab have some problem because of the port

We never used port 22 for SSH, we never had issues, but now the problem is to use HTTP.

In all case, the SSH error you are experiencing with permissions as nothing to do with GitLab.

@d-kopriwa thanks for your input. I just discovered this https://github.com/IlyaSemenov/gitlab-ci-git-push/blob/master/git-push. I don't know why simple ssh procedure is not working but the above file is working.

@raghavgarg1257 we've been using SSH for pushing until we found out that it was possible to do it with a personal access token. (Thanks to this answer https://stackoverflow.com/questions/50553049/is-it-possible-to-do-a-git-push-within-a-gitlab-ci-without-ssh/50763046#50763046)

I personally changed all our SSH keys with it, because this is the strategy chosen by gitlab-ci by default.

All you need to do is replace the ci token with yours. If you found a SSH pair a better solution, you should use it.

Edit: also, my script is shorter and more flexible than he's, it also work and doesn't need to change the git strategy used by gitlab.

All you need to do is to add the content in a script file and execute it in before script, then you will be able to push normally if you set the GL_TOKEN with your personal access token.

@d-kopriwa I suppose deployer-token:$DEPLOYER_TOKEN is the new gitlab user we need to create and I cannot do that in my case as I mentioned earlier.

I am not allowed to create a new account for a runner because that entails making a real email on our server and doing many related task to it.

Yeah, that answer seems like a nice workaround, but I am going to stick with SSH keys only. Thanks for all your help.

@raghavgarg1257 if you look at my answer, there is no deployer-token:$DEPLOYER_TOKEN.

Also, not that when you use a token, the value used for the username doesn't matter at all.

A personal_access_token is owned by user, so does the SSH_PRIVATE_KEY so your report doesn't make sense IMO.

I think people should not use the SSH_PRIVATE_KEY which expose an ssh access to your host.

This worked for me

  before_script:
    - url_host=`git remote get-url origin | sed -e "s/https:\/\/gitlab-ci-token:.*@//g"`
    
    # runner runs on a detached HEAD, checkout current branch for editing
    - git reset --hard
    - git clean -fd
    - git checkout $CI_COMMIT_REF_NAME
    - git pull origin $CI_COMMIT_REF_NAME

    - git remote set-url origin "https://$GIT_ACCESS_USER:$GIT_ACCESS_PASSWORD@${url_host}"
    - git config user.name $GIT_ACCESS_USER
    - git config user.email $GIT_ACCESS_EMAIL

@d-kopriwa

so does the SSH_PRIVATE_KEY so your report doesn't make sense IMO.

SSH keys are not associated with any account, that's the beauty of it. I will just create a key pair on my local machine, upload private key to repo's ci secret variable and public key to deploy keys and deleted it from my machine. So now only people with Master access to the repo and can do harm.

Also if I create personal-access-token with my account, and decide to leave the company, my account will be deactivated and they won't be able to deploy anymore. This doesn't seems like a good approach to me.

Btw, Thanks for the new info, I didn't knew this.

Also, not that when you use a token, the value used for the username doesn't matter at all.

@raghavgarg1257 that's true, I didn't think about the deploy keys feature that's why.

Probably because it requires to configure group or repo and require more work to replace keys on all the repos.

We preferred a single secret that allows the downloads of keys from a http service on the intranet, but we have a user for that and we do not care about creating mailbox for robots.

I still hope they will solve this as this at some point.

While the design decision on this issue isn't made, I'd like to share a snippet that uses admin access to a gitlab server, sets up a bot user with developer access to a project, and adds an impersonation token of this bot to the project variables. Then the repository may be set up to push using that token.

Here's the code, using python's wrapper for gitlab API: https://gitlab.kwant-project.org/snippets/275

Feedback welcome.

@anton-akhmerov I like this idea. We actually think to use simple release bot for this purpose and sign it as maintainer to groups. As only Contributer can't push to master branches (protected).

But Im wondering of this will support GPG & makes users as deployment commit name owners

@david.mark that all depends on what you do in the container, so you'll need to pass the private key as a secret variable, and correctly configure git to specify the person making the commit.

@anton-akhmerov using "bot" users was discussed all the way at the beginning of the thread as a "workaround" for not being able to impersonate the user that is triggering the CI job when interacting with the projects git repo. In general the approach will do (half) the job but when you are on a pay-for version of Gitlab you will have to pay for every bot user and also there is no way for you to allow different level of access for different branches unless you create more "bot" users which in turn will rake up more licences.

@nielsbne I realize that using a bot user was discussed, and I also realize that it is not the correct in some circumstances, and that even when it works it is merely a workaround.

I shared the bot creation snippet in hope that it might be useful to some, and I apologize if my message left other possible interpretations open.

@teemo https://gitlab.my.salesforce.com/0016100000dG6oO

added direction label

mentioned in issue TeamACube/HouseOfIgnotus#44

added to epic &453 (closed)

added [deprecated] Accepting merge requests label

added 1 deleted label and removed 1 deleted label

added Category:Continuous Integration label

changed milestone to %12.2

changed milestone to %12.3

Description from https://gitlab.com/gitlab-org/gitlab-ce/issues/53558:

It is very useful in a variety of situations for a CI job to push commits back to the repository, but it isn't easy to setup. It can be done either with SSH using deploy keys or HTTP using personal access tokens, but it isn't easy to setup.

GitLab should make it possible to push back to the repository by simply toggling a checkbox or adding an option to the .gitlab-ci.yml file.

Further details

Some use cases:

automatically fixing formatting and linting errors - tools like Prettier can automatically fix issues. On the lint job failing, it should be easy to automatically commit the fixed files and push the new commits, thus triggering a fresh pipeline on the new commit.
automatically creating new branches – using a scheduled pipeline I can pull data from an external system to update dependencies, draft blog posts or release notes. It should be easy to commit the content that has been generated and open a merge request (albeit - opening the merge request is a separate problem)

Proposal

Add a button to automatically configure deploy keys for a GitLab CI and configure the known hosts so all I have to do is set the committer name/email, commit and push.

What does success look like, and how can we measure that?

(If no way to measure success, link to an issue that will implement a way to measure this)

Links / references

mentioned in issue gitlab-com/www-gitlab-com#3384 (closed)

added [Deprecated] Category:Runner label

mentioned in epic &573 (closed)

I'm having a hard time believing this request to support basic CI/CD functionality has been open for THREE YEARS? Really?

Are there any plans to support this? We're looking for native support as the currently proposed solutions are a bit hack-y for such a simple task.

One common scenario would be auto-incrementing the version of number of our mobile apps with each build and committing the changes.

I'd like to add a proposal for how to do this:

A CI variable exposed (to the container), that contains the private component of an RSA key-pair. The public component would be internally added in GitLab as a "Deploy Key" into the project. The key could then be easily used in the container:

echo "${GIT_COMMIT_KEY}" > /root/.ssh/id_rsa
chmod 600 /root/.ssh/id_rsa
...
git commit ...
git push ...

From a user point of view, it would be how CI_DEPLOY_PASSWORD works for pushing to the registry from CI.

This would be more secure, as well, since the keypairs could be auto-rotated and ephemeral.

There should be a CI variable exposed, that contains the private component of an RSA key-pair.

This sounds simple and secure. +1!

@jacobvosmaer-gitlab mentioned some security concerns on gitlab-ce#56692. @jacobvosmaer-gitlab can you review the various proposals here and provide your insight into them and if any of them increase or decrease your concerns.

@brendan this is a long issue with lots of different people talking. Which proposals do you want me to review exactly?

@jacobvosmaer-gitlab @brendan

maybe gitlab-ce#41084 could be a good start and could kill a lot of flies at once! Especially since 11.7 to post release urls as well!!!

@brendan my concern about allowing blanket pushes from CI is that it's very permissive. If the user makes a mistake in their CI script they can end up pushing to the wrong branch. If the push token leaks into the build log, then whoever sees that can push anything they want to the repository.

It's not just security, it's also about not giving users a "footgun".

I'm not really an expert on any of this but this is what I was thinking of when I wrote https://gitlab.com/gitlab-org/gitlab-ce/issues/56692#note_133680746

@jacobvosmaer-gitlab would moving the push configuration to .gitlab-ci.yml be a possible way to reduce the chance of mistake or security risk?

Something like

job:
  script:
    - git commit ...
  push_to:
    branch: master

That's the sort of things that reduce the footgun factor, yes.

You shouldn't be asking me though. I am not a CI expert, nor an application security expert. I work on Gitaly and I don't think these feature ideas have anything to do with that.

Apologies, I addressed you merely because of the menion by @brendan

@jacobvosmaer-gitlab Regarding footgun risk, I'd argue that pushing to repositories and registries is already done via CI jobs and that pushing to git has similar risk.

One thing I've seen in other systems is that deploy keys are controlled like users are. If you have a protected branch, you can create an exception for the deploy key by name (similar to a user). That way deploy keys are less risky.

@anton-akhmerov I like your concept - especially as it relates to the ability to use protected runners/tags. If one gave permission to a known runner - which was protected and didn't run arbitrary code then it would be a much safer workaround.

changed epic to &573 (closed)

Adding my use-case here to show why I want this feature.

Repositories

Repo 1: gRPC proto definition files
Repo 2: ruby bindings generated from gRPC definition files.
Repo 3: java bindings generated from gRPC definition files.
Repo 4: ruby gRPC server that uses the bindings in Repo 2

** Current Process **

Committer adds a commit that updates the proto definitions in Repo 1 with commit message "Add definitions for Customer Message"
CI workflow runs and lints the definitions in Repo 1 then triggers builds in Repo 2 and Repo 3
Repo 2 workflow generates the ruby bindings
Repo 2 workflow publishes the ruby bindings to a gem server.
Repo 2 workflow triggers a build in Repo 4 that uses these new bindings
Repo 3 workflow generates the java bindings
Repo 3 generates the ruby bindings and pushes to a maven server.

The problem here is that nowhere in gitlab git repos do I have the generated ruby and java code. It only exists in the published packages to gem and maven. If I want to have the generated bindings in gitlab I have to do it myself manually on my local machine and push the changes up

** Desired Process **

What I want to happen is.

Repo 2 workflow generates the ruby bindings
Repo 2 workflow commits the updated ruby binding code to the repository
Repo 2 workflow publishes the ruby bindings to a gem server.
Repo 2 workflow triggers a build in Repo 4 that uses these new bindings
Repo 3 workflow generates the java bindings
Repo 3 workflow commits the updated java binding code to the repository
Repo 3 generates the ruby bindings and pushes to a maven server.

When I commit the code in steps 4 and 7 I want to use the same commit message and committer ID as the original commit to in Repo 1

You can see most of this in https://gitlab.com/ganges/grpc-ruby-code/blob/experimental/.gitlab-ci.yml

I also agree with @adam.pryce that the footgun risk isn't really greater than what we can already do with regards to docker registries etc.

Why do you have repos 2 and 3 instead of just generating the bindings as an artifact of repo 1?

Since I created this issue almost 3 years ago a lot has changed. For our projects where we were in need of this feature we have switched to a mono-repository structure where projects with inter-project dependencies are developed on. Probably this is also a solutions for some other teams/developers. Of course this comes to the cost of access management and some other parts as it is just one repository with a logical structure within.

For further reading I suggest Lerna (JS) and split.sh/MonoRepoBuilder (PHP).

@tomelliff I don't want to detail this too much and go off-topic but Repos 2 and 3 are where I commit (and would like to be able to commit as part of the pipeline process) the language binding code.

I can't believe so simple thing like "put a tag on some commit after successful build" still can't be implemented on GitLab, just because runners for some reason can't push to repository (even only tags!). Please, implement this feature.

changed milestone to %Backlog

@brendan It's disappointing that this has changed to Backlog. Yes, it keeps getting pushed back, but changing to Backlog says to everyone watching this issue that it's no longer on the roadmap for the foreseeable future. There have been plenty of suggestions as to how to implement this securely, and it's a feature watched by a lot of people, including multiple "customers".

Can't agree more, especially without explanation why this has been pushed back. Other vendors like Bitbucket Pipelines has this feature already implemented, why not GitLab?

+CircleCI

Thanks all for the feedback. While I agree that this issue is still valid, we have chosen not to prioritize yet because:

It does not fit entirely in our ~"Product Vision 2019" for this year to prioritize this issue over others that we have focused on - you can learn more about that here: https://about.gitlab.com/direction/verify/
Many workarounds have been mentioned on this issue. While I don't believe any of those workarounds are "better" than first-class support for this feature, I believe that the ability to push from CI is something that is a "with great power comes great responsibility" situation and thus whatever the solution is should be clear to the user about this.
Given that power/responsibility balance and the possibilities of security and footgun risks discussed above, we would need to have a convincing story of how we compensate for those before we decide to implement this. As pointed out it is possible such a solution exists, but I would love to see that discussed in an MR adding that feature or a more concise proposal issue linked to this one.

Even given all that, I haven't decided to close the door on offering this support in the future. If it was my opinion that we were not going to accept this ever, I would have closed the issue. However, given the popularity and the fact that I'm not convinced we shouldn't have first-class support for this, I chose to simply put it in our %Backlog (meaning GitLab, Inc. currently doesn't prioritize it over other items that are scheduled) but leave gitlab-ce~1890178 so that we would accept proposed changes from the community around this feature.

@brendan Except, that according to the direction document, items move from no milestone, to backlog, and then to having a milestone added (https://about.gitlab.com/direction/verify/#prioritization-process). This issue is going backwards in the process, not forwards.

Surely, Gitlab should be focusing on things that have already been allocated milestones, especially ones that have had milestones pushed continuously for 3 years.

You can argue that it shouldn't have had a milestone added in the first place, but maybe Gitlab should focus on clearing the "backlog" of issues that have already been issued milestones, and backlog new issues until some of the older issues have been solved.

marked this issue as related to gitlab-runner#256 (closed)

A current paying customer is asking about this (ZD internal only). They want to push tag back to the repo from GitLab runner.

@brendan Do we have any particular recommended for the workaround for this?

Workaround: Using a special user with repository_write or api token.

changed milestone to %Backlog

removed Category:Continuous Integration label

Apologies for the noise here. This, along with five or so other issues was inadvertently moved. Moving it back to the correct project.

changed milestone to %Backlog

moved from gitlab-runner#4421 (moved)

added grouprunner label

changed the description

Hey everyone, wanted to check in on this issue to see if we can find a simple solution that can be scheduled that solves most of the use cases. Reading through this and related issues (in particular https://gitlab.com/gitlab-org/gitlab-ce/issues/41084), one simple proposal that might be a solid step forward is:

Protected runners get an access token that is equivalent in permissions to the user who triggered the pipeline, but points back to the pipeline as the executor rather than the user so that context that this was performed by automation is not lost.
The token can then be used to push to the repo, access API, interact with the registry, and so on. Basically, anything the triggering user can do.
This is an optional feature and would be off by default.
By leveraging protected runners/branches/tags, we give the user control over when this more powerful token should be available (in addition to simply turning the entire feature off or on.)

@ayufan @estrike would appreciate your thoughts also.

Thanks @jlenny for the summary and proposal.

The Package team is trying to work through this issue as well, since it is a very common use case for users to build an image via CI. However, we currently do not allow them to untag/remove the image. As a result, the UI for the container registry is slower and more cluttered and storage costs are inflated.

For the Container Registry and for the various package manager integrations, I think we should allow CI_REGISTRY_USER to delete images/tags/packages according to their user permissions. We are attempting to tackle that now with https://gitlab.com/gitlab-org/gitlab-ce/issues/40096. Although if we create a new token, we should ensure they have permissions to delete images/tags from the container registry as well.

For other actions, like pushing to the repo, I like the idea of an additional token that leverages the users permissions as this is in-line with our existing CI permissions model. I'm not against starting with protected branches or with the feature off, but I suspect that we will quickly receive requests to support the new token for non-protected branches as well.

@ggelatti - Do you have any additional thoughts having started on gitlab-ce#40096

@trizzi Adding extra token/permissions for the build should be fine. Not sure about the other issues mentioned like triggering infinite loops, for example.

@jlenny Could you change your post to number your points? I will refer to them in the order you posted them for now...

1> I think that's reasonable, it's not really the person that's running it, but you're right, we would want the same permissions as the person who triggered it, whether through a tag, commit, or trigger, pressing a play button, or whatever.

2> No comment

3> Off by default but configured by who? sys-admin/owner of pipeline?

4> Do you mean it would only be available on a "protected" branch, runner, or tag?

Thanks.

I don't want this feature off by default because I don't want to have to do anything but drop a yml file into the repo to make the build work.

That's a good point. Maybe, if @jlenny is considering off by default due to a security concern it could instead have some sort of permission, and that permission is implicitly given to certain groups?

Added numbers above, thanks for the suggestion @trenton.d.adams.

3> I'm open to it potentially being on by default, but would like someone from our security team to review. Given it would only be available to protected runners maybe it is fine, but it could be an unwelcome surprise. I imagined project or group owner would turn it on.

4> Yes, that's what I'm thinking. Since this is a more sensitive credential, which protected tags/branches/runners are designed to protect.

@jlenny I can see it being enabled on all branches as a possible thing people might want. Would it be possible and easy for that to be configurable, as to whether it's only protected branches?

I personally like all branches to work the same way. Mainly because I can then try things with my pipeline while I'm doing development. Another reason is doing a quick production patch from a branch (which might not normally occur, depending on one's workflow) to make it quick and easy to go live. This would particularly be useful in the case where people adopt real CI master branch based development but want to quick patch something.

I'm not a huge fan of the protected runners/tags/branches limitation - in our use case, we would utilise it for CI-identified remediations (in Drupal, getting a clean config export committed to code, or bringing a configuration back within a predefined security envelope) - but we would be less willing to conduct this on a master (protected) branch. Making it more configurable would be a preference - we currently do something similar, but utilise a shared, generic account to commit output logs back into the repo after an audit run.

I'd agree here, we're an EE customer and we currently force our users to test on non protected branches/runners because we use protected runners to isolate our production runners for deployment jobs.

Saying this is only for protected runners breaks the ability to test this outside of the very systems designed to allow this "safe environment"

assigned to @jlenny

removed [deprecated] Accepting merge requests label

changed milestone to %Backlog

ZD: https://gitlab.zendesk.com/agent/tickets/132452 (internal)

unassigned @jlenny

added [deprecated] Accepting merge requests label

mentioned in issue #20416

Everyone, given the protected runners limitation discussed above doesn't really work, we're looking at a universal solution around job token permissions to unlock this along with several other issues in one go, and in a consistent way. The idea is that there will be a project setting that allows you to finely control CI_JOB_TOKEN permissions. There is a brief thread at #20416 (comment 216069060) that it would be great to have your feedback in, especially if you feel that does not meet your use case.

Everyone, I've created and scheduled #35067 (closed) to be the implementation issue for this effort. I've summed up the state of discussion there in the description and would appreciate your review and feedback. The reason for creating a new issue is that the plan and problem to solve have been generalized, and reworking this issue with older comments could be confusing. The intention though is that the new issue completely (and in a more general, powerful way) meets the use case here.

As such, I'm marking this as a duplicate and closing this one. If you feel I've missed something please do let me know in that new issue - we can reopen this one, adjust that one, or whatever is needed to make it right.

mentioned in merge request morley-framework/morley!71 (merged)

Opened 3 years

The birth is heavy. I look forward to a happy ending.

marked this issue as related to #35067 (closed)

marked this issue as a duplicate of #35067 (closed)

closed

removed milestone

I just stumbled across this thread because I were not able to push changes back to the repository. My scenario is a release management for a TypeScript project with lerna so merges to master gets tagged and version bumps automatically (own runner with docker executor). I tried the GitLab Access Token (Profile Settings > Access Token) with the api and write_repository scopes. After some failed attempts I managed to get it to work properly:

stages:
    install

image: matzeeable/ci-php-composer-node:latest

variables:
    # Allow git interactions (e. g. lerna version & publish)
    GIT_AUTHOR_NAME: $GITLAB_USER_NAME
    GIT_AUTHOR_EMAIL: $GITLAB_USER_EMAIL
    GIT_COMMITTER_NAME: $GITLAB_USER_NAME
    GIT_COMMITTER_EMAIL: $GITLAB_USER_EMAIL

install:
    cache: {}
    stage: install
    script:
        # Leave detached git HEAD
        - git checkout -B "$CI_COMMIT_REF_NAME" "$CI_COMMIT_SHA"
        # Create stageable file and stage it
        - echo "from-ci" > from-ci.txt
        - git add -A
        # Commit and tag (note the [skip ci] to avoid loop)
        - 'git commit -m "chore: release [skip ci]"'
        - 'git tag from-ci -m "chore: release [skip ci]"'
        # Push back to repository
        - git push --follow-tags --no-verify "https://gitlab-ci-token:$GITLAB_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git" "HEAD:$CI_COMMIT_REF_NAME"

This means I don't have to store a plaintext password in my variables, but can use the default access token from GitLab. For this, I have added the GITLAB_TOKEN variable in my CI/CD variables (Project settings).

To lerna users refer to this issue but if you want a short answer how to use lerna in GitLab CI, use the following script:

variables:
    GL_TOKEN: $GITLAB_TOKEN

    # [...]
    script:
        - git remote set-url origin "https://gitlab-ci-token:$GITLAB_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git"
        - lerna publish --yes

Possibly a GitLab Runner Bug?

Instead of building the push URL by ourself (as you can see in the above solution) I tried the predefined variable for that kind of URL. I just realized that I can not rely on $CI_REPOSITORY_URL because the following failed:

$ git push --follow-tags --no-verify "$CI_REPOSITORY_URL" "HEAD:$CI_COMMIT_REF_NAME"
remote: You are not allowed to upload code.
fatal: unable to access 'https://gitlab-ci-token:[MASKED]@gitlab.com/xxx/xxx.git/': The requested URL returned error: 403

My guess is that the runner never receives the [MASKED] placeholder correctly. So I need to stay with the above solution.

I still want to use `origin`!

EDIT: I just found a way to still rely on origin. Just do the below before you push anything to your repository:

git remote set-url origin "https://gitlab-ci-token:$GITLAB_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git"

See the above lerna example.

Thanks so much for the above @matzeeable . I have a similar scenario deploying a package to NPM and wanting to automatically bump the version in package.json via CI when a tag is created from the Gitlab UI to mark a release. This saved me so much time

I do think it'd be better DX if runners could have push enabled without all this though ;-)

@kane.clover Can you share that example? @sahbabou is working on a new gitlab-ci.yml template for node.js projects and we have been blocked on bumping the version in package.json when a tag is created.

This is what we have. It does bump the version and commit the updated package.json The hiccup is using a token to push and not the password/PAT

- echo "unsafe-perm = true" >> .npmrc
    - npm version minor --git-tag-version=false
    - npm publish --verbose
    - git add package.json
    # You can add a getName and getVersion key under scripts in your package.json to return the name and version of your package and add it to the commit message
    #"scripts": {
     #   "getVersion": "echo  $npm_package_version",
      #  "getName": "echo  $npm_package_name"
    #  },
    - git commit -m "[ci skip] update $(npm run getName -s) version to $(npm run getVersion -s)"
    - git push -o ci.skip "https://sahbabou:${PASSWORD}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}" "HEAD:${CI_COMMIT_BRANCH}"

Sure :-)

publish-npm:
  image: node:12.16.1
  stage: deploy
  only:
    - tags
  before_script:
    - npm install
  script:
    - env
    - echo $CI_COMMIT_REF_SLUG
    - echo $CI_COMMIT_TAG
    - git config --global user.name "$GITLAB_USER_NAME"
    - git config --global user.email "$GITLAB_USER_EMAIL"
    - git remote set-url origin "https://gitlab-ci-token:$GIT_ACCESS_TOKEN@$CI_SERVER_HOST/$CI_PROJECT_PATH.git"
    - echo '//registry.npmjs.org/:_authToken=${NPM_TOKEN}'>.npmrc
    - echo "unsafe-perm = true" >> ~/.npmrc
    - npm --no-git-tag-version version $CI_COMMIT_TAG
    - 'git commit -am "chore: release $CI_COMMIT_TAG [skip ci]"'
    - npm publish --access public
    - git push origin HEAD:master

Thank you!

/cc @sahbabou

thank you @kane.clover @trizzi

mentioned in issue #35067 (closed)

mentioned in epic &3559

Feature Request: Allow runners to push via their CI token

Proposal A:

Example:

Proposal B:

Example:

Possible problems:

Designs

Child items ...

Activity

Further details

Proposal

What does success look like, and how can we measure that?

Links / references

Possibly a GitLab Runner Bug?

I still want to use `origin`!

Feature Request: Allow runners to push via their CI token

Proposal A:

Example:

Proposal B:

Example:

Possible problems:

Relates to

Activity

Further details

Proposal

What does success look like, and how can we measure that?

Links / references

Possibly a GitLab Runner Bug?

I still want to use origin!

I still want to use `origin`!