cannot access protected variables when using `only: merge_requests` parameter

There is a workaround.

You can set job to run always, but not on empty merge request title:

job-name:
  only:
    refs:
      - branches
      - master
      - tags
  except:
    variables:
      - $CI_MERGE_REQUEST_TITLE =~ /^$/
  script:
    - echo $protected_variable_name

You need to do that everywhere you used only: merge_requests before. As soon as there is one only: merge_requests parameter present in any job in a pipeline only these jobs will run on merge requests.

I see it's documented like that, but why is the default behavior set to not run on merge requests for any job that doesn't have only: merge_requests configured as soon as any job in the pipeline uses this?

This is very counter intuitive in my eyes. As this is a job setting it should only affect the job it is set in and not the other jobs in a pipeline.

I can think of many cases where I have several jobs in a pipeline I always want to run, but only some to run on merge requests only. With the current behavior I need to add the only: merge_requests parameter for all jobs in the pipeline I want to run always.

~~Now even the simple job which just prints variables without any additional settings is not able to print the protected variable. This works on our company GitLab EE version 11.9.4.~~

Protection for that branch was switched off somehow. Switched it back on and behavior is as described above and the workaround seems to work.

This issue will be moved to the Community Edition issue tracker because it relates to a feature that is also available in Core

added devopsverify pipeline typebug + 1 deleted label

added CI variables label

added Category:Continuous Integration label

Two releases went by and the bug is still there. Is anybody looking into this?

This really is a very annoying issue.

We would like to push some artifacts to a testing channel on an Artifactory instance on a merge request, but we can't because we can not access the protected access token we don't want to expose to anyone.

@drewcimino can you see if you can understand what's going on

added priority2 severity2 labels

It seems to me that this is by design. Merge Request pipelines are detached, and thus not associated with any protected ref and given access to protected variables.

If I had access to a project but not the production token variables on our protected master, I would be able to push a new feature branch, open an MR to master, and dump all of our production token variables values to a job log.

@tobias.hartmann78 The first workaround (off the top of my head) would be creating a second, protected, preproduction branch to merge your changes into to as a step in between your feature branch and master. The functionality of:

only:
  - merge_requests

would be more like

only:
  - variables:
    - "$CI_COMMIT_REF_NAME == "preproduction"

Let me know if that's possible or not, or if I'm misunderstanding what you're trying to do.

changed milestone to %Backlog

added [deprecated] Accepting merge requests label

Hi @drewcimino , you understood well @tobias.hartmann78 's request. Thank you for looking into it and for the workaround. Actually, that would be a bit difficult to ask everyone to create a pre-prod branch and also to do it for every single project that is using a protected environment variable. Is there a way that Merge Request pipelines become 'not detached'?

Hi @rzaddem!

Regarding the detachment: I don't think so. Merge Request pipelines are run on the result of merging the source branch into the target - this is a new commit that Gitlab creates. To avoid a detached Pipeline, Gitlab would be effectively auto-merging developer MRs into protected branches.

My suggestion isn't that everyone make a preproduction branch. If everyone could, it would mean they all have been granted protected-ref access. My suggestion is a process solution that would involve someone currently with protected ref access to creating and maintaining a separate protected branch like they would a staging environment.

It's worth noting that if a given person or entity can trigger a build that accesses protected variables, they have access to the value of your protected variables. Masking can keep the values out of your job logs, making the logs a bit easier to share, but nothing can stop a build script from sending available variables off to any third-party endpoint, or anything else written into a job script. There's no way around this. It's a conceptual problem and not one of implementation, so I'm not sure there's anything we can do in Gitlab to change it.

Hello @drewcimino,

sorry, that I come back to this that late.

First regarding the security concerns with protected variables: We are aware, that using protected variables does not completely ensure their security, but it at least in some way obfuscates them.

Your suggestion of moving to a model with an extra development branch as a step between the feature branches and master certainly has the potential to solve the problem with the credential visibility, but it would mean for us to adjust all our processes which I currently don't see.

To save build time we usually limit builds to merge requests, and the resulting commits to release branches and master. We use the merge requests to review, and finalize features which involves producing packages and testing them. To do this we would need access to credentials to make the produced packages available for testing.

I was hoping it would be possible to implement a fix to just somehow propagate the variables to the MR pipelines if they are started from a merge request on a protected branch.

Best regards Tobias

@tobias.hartmann78 Is this a private repository? If so, the answer for your team might be to not use protected variables. The setup you're looking for would leave the values fairly exposed to anyone really looking for them, so it might make more sense to just protect the refs so people can't accidentally merge. Propagating protected variables from a protected ref to a non-protected ref (i.e. the Gitlab merge commit) is effectively un-protecting the variables anyway. You can still mask them in job logs.

Rather than

packaging-and-testing-job:
  only:
    - merge_requests

you would use a configuration like:

packing-and-testing-job:
  only:
    variables:
      - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "my-protected-branch" # || $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /-feature$/, etc. etc.

@drewcimino

Actually we currently use the variable evaluation features and regular expressions as a workaround, but maintaining this means setting up conventions and regulations that wouldn't be necessary if we could just use only: merge_requests.

As the merge_request context is only implicitly created by GitLab CI automatically, and needs to be created from a protected branch anyway to have access to the variables, propagating them wouldn't change anything regarding variable access. It is just counter-intuitive that the merge_request context is different from the usual branch context the merge request is created on.

But as I see this was added to a milestone I am looking forward to a solution for this. Would be nice if we can get rid of the more complex variable evaluation.

added grouppipeline execution label

changed milestone to %12.7

@matteeyah What do you think of considering a merge request ref to be protected if the merge request source and target branches are both protected? I'm looking at Project#protected_for(ref) and thinking we could detect a merge request ref, and in turn check the branches involved in the MR.

Security concerns:

Does this open up a vulnerability by allowing someone without protected branch/variable access to create a merge request between two protected branches? Even if the Pipeline runs, someone with protected ref access would have had to commit to one of the branches, so this seems to be safe.
Does merely running a Pipeline on protected refs become a vulnerability? Pipelines can trigger dangerous actions (e.g. deploys, deletions) that are restricted to running on protected refs. Or is this a separate issue of protected environments? If so, a remedy would create a contingent permission check where we block an MR creation with protected branches if the creator doesn't have protected ref access. This could get funky and/or bad.

@tobias.hartmann78 Would having MRs with protected source and target branch refs have access to protected variables help you?

Hello @drewcimino that would help a great bit. We were expecting the variables to be accessible on merge requests for protected branches only anyways.

Today I ran into exactly the scenario described in #1. My intuition would be that a MR between two protected branches is, itself, protected

@drewcimino Can someone without write access trigger a pipeline on an already existing commit?

I'd say that the answer to your second question is yes - running pipelines is inherently dangerous.

changed milestone to %12.7

added SLOMissed label

changed milestone to %12.8

added missed:12.7 label

added backend workflowplanning breakdown + 1 deleted label

Due to the volume of ~P2 items that the team is working through, this item is moving to the backlog and the team will pull in top priority P2s as fast as they can into each milestone planning. This will avoid the situation of P2 issues moving from milestone to milestone, but if you feel this particular P2 is of urgent import please @ me to let me know and we can look at fast-tracking it.

changed milestone to %Backlog

removed workflowplanning breakdown + 1 deleted label

Silver customer experiencing this issue: https://gitlab.zendesk.com/agent/tickets/149410 (Internal)

added customer label

Silver customer experiencing this issue: https://gitlab.zendesk.com/agent/tickets/161036 (internal use)

added Category:Secrets Management label

added sectionops label

added 1 deleted label and removed grouppipeline execution label

added devopsrelease [DEPRECATED] label and removed devopsverify label

removed SLOMissed label

added SLOMissed label

added groupconfigure [DEPRECATED] label and removed 1 deleted label

Bronze user also experiencing this issue

added devopsconfigure [DEPRECATED] label and removed devopsrelease [DEPRECATED] label

added priority4 label and removed SLOMissed priority2 labels

added SLOMissed label

cannot access protected variables when using `only: merge_requests` parameter

Summary

Steps to reproduce

Example Project

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Possible fixes

Designs

Child items ...

Activity

cannot access protected variables when using `only: merge_requests` parameter

Summary

Steps to reproduce

Example Project

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Output of checks

Possible fixes

Relates to

Activity