Skip to content
GitLab Next
  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • GitLab GitLab
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 43,809
    • Issues 43,809
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1,416
    • Merge requests 1,416
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.org
  • GitLabGitLab
  • Issues
  • #35438
Closed
Open
Created Nov 01, 2019 by Ben Prescott @bprescott_↙ ☺@bprescott_🌇Developer

Backend: Regexp CI/CD variables are evaluated as strings

Summary

When you have a Regexp variable, you cannot use it as a Regexp in rules/only conditions.

For example;

variables:
  teststring: 'abcde'
  pattern: '/^ab.*/'

test1:
  script: exit 0
  rules:
    - if: '$teststring =~ $pattern'

test2:
  script: exit 0
  rules:
    - if: '$teststring =~ /^ab.*/'
  • The test1 job is not created because the backend makes string comparison between "abcde" and "/'^ab.*/".
  • The test2 job is created because the backend makes regexp comparison between "abcde" and /'^ab.*/.

Customer background

  1. Customer ticket (internal) was around the fact that any pattern definition in a variable containing .* could not be made to work, having tried several protection strategies to define and / or use the variable

    • I might have found a further misbehaviour, which is that while /^ab/ matches abcde when applied direct, it doesn't work when matching via a variable. My further test case of /^ab.*/ probably doesn't work because /^ab/ doesn't.
  2. (now in #36940 (closed)) Jobs do get created when the regex (in a variable) contains $ but when that variable is exported to the runner (and in the runner debug output) the $ is missing

edit 2019-11-21 - title changed, and content largely hidden for the issue relating to $, now: #36940 (closed)

Steps to reproduce

The example project contains a full CI configuration; this is a summary.

These are the test patterns and test string.

variables:
  teststring: 'abcde'
  pattern1: '/^abcde$/'
  pattern2: '/^abcd$/'
  pattern3: '/^ab.*/'
  pattern4: '/^ab/'
  pattern5: '/^abcde.*/'
  pattern6: '/^abcde/'
  pattern7: '/^.*$/'
  pattern8: '/.*/'
this section relates to the $ issue, which has been put in a separate issue

see: #36940 (closed)

In respect of $ one of my example jobs dumps the shell environment. I forgot to enable debugging. I found it originally using CI_COMMIT_DESCRIPTION, and this is what I captured:

Author: Ben Prescott <bprescott@example.com>
Date:   Fri Oct 25 15:16:14 2019 +0100

    ci: add a description
    /^mo$/
++ export 'CI_COMMIT_DESCRIPTION=/^mo/'
++ CI_COMMIT_DESCRIPTION='/^mo/'

Reverting to the example I made for this ticket, if you look at the dumped variables in the output/screenshot section below, all of the variables containing $ have had it removed.

Moving on from $ to the pattern matching issue ..

Defining job the following way, using only, seven jobs get created. pattern2 fails as expected.

Job definition where pattern match is applied direct
direct5:
  stage: pmdirect
  tags:
    - shell
  script:
    - /usr/bin/true
  only:
    variables:
      - $teststring =~ /^abcde.*/

Via variables, only three patterns work - 1, 5, and 6. I've tried it with both only and rules

Job definition where pattern match is applied via variable
variable1:
  stage: pmvariable
  tags:
    - shell
  script:
    - /usr/bin/true
  only:
    variables:
      - $teststring =~ $pattern1
#
varrule1:
  stage: pmvarrule
  tags:
    - shell
  script:
    - /usr/bin/true
  rules:
    - if: '$teststring =~ $pattern1'
      when: on_success

I've also created a set of variables in the GUI, to explore whether reading the regexes from the yaml is where it goes wrong.

Same result.

Example Project

yaml here

Note: I'm testing on a standalone install, and my project is configured to expect the CI yaml definition in ci/definition.yml

This should show as commit edccc5dd which matches screen shots etc. below.

What is the current bug behavior?

  1. Jobs that get created when a pattern match rule works explicitly fail to get create when the pattern is stored in a variable instead. Patterns including .* don't work, and this additional /^ab/ case which I only just found, and am not sure what the general case is.

  2. (now in #36940 (closed)) $ doesn't appear in the debug output and exported to the runner. Issue was noticed investigating the other issue.

What is the expected correct behavior?

  1. The same jobs that get created when pattern matched explicitly should get created when the patterns are in variables.

  2. (now in #36940 (closed)) Bearing in mind this is the only debug mechanism .. strings make it through verbatim. It may well be the case this simply isn't possible, because $ has a specific meaning in shell! There might be times when customers don't expect it to be missing. For example .. commit messages, or any other variable that could contain any valid character.

Relevant logs and/or screenshots

Output from example project.

CI screenshot showing which jobs create

variables set via GUI

Expand for output of environment dump
Running with gitlab-runner 12.3.0 (a8a019e0)
  on localrunner J2yXYmYx
Using Shell executor...
Running on bprescott-gitlabtest-0...
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction/.git/
From https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction
 + 0ffbfd0...edccc5d 20191101-35438 -> origin/20191101-35438  (forced update)
Checking out edccc5dd as 20191101-35438...
Skipping Git submodules setup
$ /usr/bin/env | /usr/bin/sort

CI_API_V4_URL=https://bprescott-gitlabtest-0.do.gitlap.com/api/v4 CI_BUILD_BEFORE_SHA=0ffbfd0089d1d32f7be2ab337310fc89a023e1b8 CI_BUILD_ID=1171 CI_BUILD_NAME=envdump CI_BUILD_REF=edccc5dd95833c908bcac8a3ea3e187a3bb7e479 CI_BUILD_REF_NAME=20191101-35438 CI_BUILD_REF_SLUG=20191101-35438 CI_BUILDS_DIR=/home/gitlab-runner/builds CI_BUILD_STAGE=debug CI_BUILD_TOKEN=[MASKED] CI_COMMIT_BEFORE_SHA=0ffbfd0089d1d32f7be2ab337310fc89a023e1b8 CI_COMMIT_DESCRIPTION= CI_COMMIT_MESSAGE=ci: compare hard coded patterns with variables - for gitlab issue 35438 CI_COMMIT_REF_NAME=20191101-35438 CI_COMMIT_REF_PROTECTED=false CI_COMMIT_REF_SLUG=20191101-35438 CI_COMMIT_SHA=edccc5dd95833c908bcac8a3ea3e187a3bb7e479 CI_COMMIT_SHORT_SHA=edccc5dd CI_COMMIT_TITLE=ci: compare hard coded patterns with variables - for gitlab issue 35438 CI_CONCURRENT_ID=0 CI_CONCURRENT_PROJECT_ID=0 CI_CONFIG_PATH=ci/definition.yml CI_JOB_ID=1171 CI_JOB_NAME=envdump CI_JOB_STAGE=debug CI_JOB_TOKEN=[MASKED] CI_JOB_URL=https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction/-/jobs/1171 CI_NODE_TOTAL=1 CI_PAGES_DOMAIN=example.com CI_PAGES_URL=http://test.example.com/zd134231-disjunction CI_PIPELINE_ID=90 CI_PIPELINE_IID=90 CI_PIPELINE_SOURCE=push CI_PIPELINE_URL=https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction/pipelines/90 CI_PROJECT_DIR=/home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction CI_PROJECT_ID=1 CI_PROJECT_NAMESPACE=test CI_PROJECT_NAME=zd134231-disjunction CI_PROJECT_PATH_SLUG=test-zd134231-disjunction CI_PROJECT_PATH=test/zd134231-disjunction CI_PROJECT_REPOSITORY_LANGUAGES= CI_PROJECT_URL=https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction CI_PROJECT_VISIBILITY=internal CI_REGISTRY_PASSWORD=[MASKED] CI_REGISTRY_USER=gitlab-ci-token CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction.git CI_RUNNER_DESCRIPTION=localrunner CI_RUNNER_EXECUTABLE_ARCH=linux/amd64 CI_RUNNER_ID=1 CI_RUNNER_REVISION=a8a019e0 CI_RUNNER_SHORT_TOKEN=J2yXYmYx CI_RUNNER_TAGS=shell CI_RUNNER_VERSION=12.3.0 CI_SERVER_HOST=bprescott-gitlabtest-0.do.gitlap.com CI_SERVER_NAME=GitLab CI_SERVER_REVISION=9dbaa740018 CI_SERVER_TLS_CA_FILE=/home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction.tmp/CI_SERVER_TLS_CA_FILE CI_SERVER_VERSION=12.3.5-ee CI_SERVER_VERSION_MAJOR=12 CI_SERVER_VERSION_MINOR=3 CI_SERVER_VERSION_PATCH=5 CI_SERVER=yes CI_SHARED_ENVIRONMENT=true CI=true CONFIG_FILE=/etc/gitlab-runner/config.toml FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=false FF_USE_LEGACY_BUILDS_DIR_FOR_DOCKER=false FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER=false GITLAB_CI=true GITLAB_FEATURES= GITLAB_USER_EMAIL=bprescott@gitlab.com GITLAB_USER_ID=1 GITLAB_USER_LOGIN=bprescott-root GITLAB_USER_NAME=Ben Prescott (root) gpattern1=/^abcde/ gpattern2=/^abcd/ gpattern3=/^ab./ gpattern4=/^ab/ gpattern5=/^abcde./ gpattern6=/^abcde/ gpattern7=/^./ gpattern8=/./ HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/gitlab-runner HOSTNAME=bprescott-gitlabtest-0 LANG=en_US.UTF-8 LESSOPEN=||/usr/bin/lesspipe.sh %s LOGNAME=gitlab-runner MAIL=/var/spool/mail/gitlab-runner OLDPWD=/home/gitlab-runner PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/home/gitlab-runner/.local/bin:/home/gitlab-runner/bin pattern1=/^abcde/ pattern2=/^abcd/ pattern3=/^ab./ pattern4=/^ab/ pattern5=/^abcde./ pattern6=/^abcde/ pattern7=/^./ pattern8=/./ PWD=/home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction SHELL=/bin/bash SHLVL=2 teststring=abcde USER=gitlab-runner _=/usr/bin/env XDG_RUNTIME_DIR=/run/user/992 XDG_SESSION_ID=c9078 Job succeeded

this section relates to the $ issue, which has been put in a separate issue

see: #36940 (closed)

Note, none of the dollars appear in the above output

gpattern1=/^abcde/
gpattern2=/^abcd/
gpattern3=/^ab.*/
gpattern4=/^ab/
gpattern5=/^abcde.*/
gpattern6=/^abcde/
gpattern7=/^.*/
gpattern8=/.*/
[..]
pattern1=/^abcde/
pattern2=/^abcd/
pattern3=/^ab.*/
pattern4=/^ab/
pattern5=/^abcde.*/
pattern6=/^abcde/
pattern7=/^.*/
pattern8=/.*/

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info
# sudo gitlab-rake gitlab:env:info

System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: unknown

GitLab information Version: 12.3.5-ee Revision: 9dbaa740018 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://[gitlab-DO-test-machine].com HTTP Clone URL: https://[gitlab-DO-test-machine].com/some-group/some-project.git SSH Clone URL: git@[gitlab-DO-test-machine].com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers:

GitLab Shell Version: 10.0.0 Repository storage paths:

  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check
# sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 10.0.0 ? ... OK (10.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK

gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 2/2 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.22.0) Git user has default SSH configuration? ... yes Active users: ... 1 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

I believe support for pattern matching was visited a couple of months ago:

  • issue: gitlab-foss#64383 (closed)
  • MR: gitlab-foss!31719 (merged)

Documentation:

Pattern matching with variables is documented with just the following line

It is possible perform pattern matching against a variable and regular expression.

Following chat with @thaoyeager I've opened #36942 (closed) around this.

An initial iteration would be to document that the feature is currently very limited; it's been working to some extent since 12.3, and is not likely to be fixed before 12.7 at the earliest. I'll explore this option in !19098 (closed)

Proposal

Force convert both sides of =~ and !~ to Regexp.

Example technical change;

diff --git a/lib/gitlab/ci/pipeline/expression/lexeme/matches.rb b/lib/gitlab/ci/pipeline/expression/lexeme/matches.rb
index 4d65b914d8d..edc61db0a2c 100644
--- a/lib/gitlab/ci/pipeline/expression/lexeme/matches.rb
+++ b/lib/gitlab/ci/pipeline/expression/lexeme/matches.rb
@@ -11,6 +11,7 @@ class Matches < Lexeme::LogicalOperator
             def evaluate(variables = {})
               text = @left.evaluate(variables)
               regexp = @right.evaluate(variables)
+              regexp = Gitlab::UntrustedRegexp::RubySyntax.fabricate(regexp) || regexp
               return false unless regexp

               regexp.scan(text.to_s).present?
diff --git a/lib/gitlab/ci/pipeline/expression/lexeme/not_matches.rb b/lib/gitlab/ci/pipeline/expression/lexeme/not_matches.rb
index 29c5aa5d753..00338886310 100644
--- a/lib/gitlab/ci/pipeline/expression/lexeme/not_matches.rb
+++ b/lib/gitlab/ci/pipeline/expression/lexeme/not_matches.rb
@@ -11,6 +11,7 @@ class NotMatches < Lexeme::LogicalOperator
             def evaluate(variables = {})
               text = @left.evaluate(variables)
               regexp = @right.evaluate(variables)
+              regexp = Gitlab::UntrustedRegexp::RubySyntax.fabricate(regexp) || regexp
               return true unless regexp

               regexp.scan(text.to_s).empty?
Edited Jan 20, 2022 by Mark Nuzzo
Assignee
Assign to
Time tracking