Skip to content
GitLab Next
Closed
Created by Ben Prescott @bprescott_↙ ☺@bprescott_🌇Developer

Backend: Regexp CI/CD variables are evaluated as strings

Summary

When you have a Regexp variable, you cannot use it as a Regexp in rules/only conditions.

For example;

variables:
  teststring: 'abcde'
  pattern: '/^ab.*/'

test1:
  script: exit 0
  rules:
    - if: '$teststring =~ $pattern'

test2:
  script: exit 0
  rules:
    - if: '$teststring =~ /^ab.*/'
  • The test1 job is not created because the backend makes string comparison between "abcde" and "/'^ab.*/".
  • The test2 job is created because the backend makes regexp comparison between "abcde" and /'^ab.*/.

Customer background

  1. Customer ticket (internal) was around the fact that any pattern definition in a variable containing .* could not be made to work, having tried several protection strategies to define and / or use the variable

    • I might have found a further misbehaviour, which is that while /^ab/ matches abcde when applied direct, it doesn't work when matching via a variable. My further test case of /^ab.*/ probably doesn't work because /^ab/ doesn't.

  2. (now in #36940 (closed)) Jobs do get created when the regex (in a variable) contains $ but when that variable is exported to the runner (and in the runner debug output) the $ is missing

edit 2019-11-21 - title changed, and content largely hidden for the issue relating to $, now: #36940 (closed)

Steps to reproduce

The example project contains a full CI configuration; this is a summary.

These are the test patterns and test string.

variables:
  teststring: 'abcde'
  pattern1: '/^abcde$/'
  pattern2: '/^abcd$/'
  pattern3: '/^ab.*/'
  pattern4: '/^ab/'
  pattern5: '/^abcde.*/'
  pattern6: '/^abcde/'
  pattern7: '/^.*$/'
  pattern8: '/.*/'
this section relates to the $ issue, which has been put in a separate issue

see: #36940 (closed)

In respect of $ one of my example jobs dumps the shell environment. I forgot to enable debugging. I found it originally using CI_COMMIT_DESCRIPTION, and this is what I captured:

Author: Ben Prescott <bprescott@example.com>
Date:   Fri Oct 25 15:16:14 2019 +0100

    ci: add a description
    /^mo$/
++ export 'CI_COMMIT_DESCRIPTION=/^mo/'
++ CI_COMMIT_DESCRIPTION='/^mo/'

Reverting to the example I made for this ticket, if you look at the dumped variables in the output/screenshot section below, all of the variables containing $ have had it removed.

Moving on from $ to the pattern matching issue ..

Defining job the following way, using only, seven jobs get created. pattern2 fails as expected.

Job definition where pattern match is applied direct 
direct5:
  stage: pmdirect
  tags:
    - shell
  script:
    - /usr/bin/true
  only:
    variables:
      - $teststring =~ /^abcde.*/

Via variables, only three patterns work - 1, 5, and 6. I've tried it with both only and rules

Job definition where pattern match is applied via variable 
variable1:
  stage: pmvariable
  tags:
    - shell
  script:
    - /usr/bin/true
  only:
    variables:
      - $teststring =~ $pattern1
#
varrule1:
  stage: pmvarrule
  tags:
    - shell
  script:
    - /usr/bin/true
  rules:
    - if: '$teststring =~ $pattern1'
      when: on_success

I've also created a set of variables in the GUI, to explore whether reading the regexes from the yaml is where it goes wrong.

Same result.

Example Project

yaml here

Note: I'm testing on a standalone install, and my project is configured to expect the CI yaml definition in ci/definition.yml

This should show as commit edccc5dd which matches screen shots etc. below.

What is the current bug behavior?

  1. Jobs that get created when a pattern match rule works explicitly fail to get create when the pattern is stored in a variable instead. Patterns including .* don't work, and this additional /^ab/ case which I only just found, and am not sure what the general case is.

  2. (now in #36940 (closed)) $ doesn't appear in the debug output and exported to the runner. Issue was noticed investigating the other issue.

What is the expected correct behavior?

  1. The same jobs that get created when pattern matched explicitly should get created when the patterns are in variables.

  2. (now in #36940 (closed)) Bearing in mind this is the only debug mechanism .. strings make it through verbatim. It may well be the case this simply isn't possible, because $ has a specific meaning in shell! There might be times when customers don't expect it to be missing. For example .. commit messages, or any other variable that could contain any valid character.

Relevant logs and/or screenshots

Output from example project.

CI screenshot showing which jobs create

variables set via GUI

Expand for output of environment dump 
Running with gitlab-runner 12.3.0 (a8a019e0)
  on localrunner J2yXYmYx
Using Shell executor...
Running on bprescott-gitlabtest-0...
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction/.git/
From https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction
 + 0ffbfd0...edccc5d 20191101-35438 -> origin/20191101-35438  (forced update)
Checking out edccc5dd as 20191101-35438...
Skipping Git submodules setup
$ /usr/bin/env | /usr/bin/sort

CI_API_V4_URL=https://bprescott-gitlabtest-0.do.gitlap.com/api/v4
CI_BUILD_BEFORE_SHA=0ffbfd0089d1d32f7be2ab337310fc89a023e1b8
CI_BUILD_ID=1171
CI_BUILD_NAME=envdump
CI_BUILD_REF=edccc5dd95833c908bcac8a3ea3e187a3bb7e479
CI_BUILD_REF_NAME=20191101-35438
CI_BUILD_REF_SLUG=20191101-35438
CI_BUILDS_DIR=/home/gitlab-runner/builds
CI_BUILD_STAGE=debug
CI_BUILD_TOKEN=[MASKED]
CI_COMMIT_BEFORE_SHA=0ffbfd0089d1d32f7be2ab337310fc89a023e1b8
CI_COMMIT_DESCRIPTION=
CI_COMMIT_MESSAGE=ci: compare hard coded patterns with variables - for gitlab issue 35438
CI_COMMIT_REF_NAME=20191101-35438
CI_COMMIT_REF_PROTECTED=false
CI_COMMIT_REF_SLUG=20191101-35438
CI_COMMIT_SHA=edccc5dd95833c908bcac8a3ea3e187a3bb7e479
CI_COMMIT_SHORT_SHA=edccc5dd
CI_COMMIT_TITLE=ci: compare hard coded patterns with variables - for gitlab issue 35438
CI_CONCURRENT_ID=0
CI_CONCURRENT_PROJECT_ID=0
CI_CONFIG_PATH=ci/definition.yml
CI_JOB_ID=1171
CI_JOB_NAME=envdump
CI_JOB_STAGE=debug
CI_JOB_TOKEN=[MASKED]
CI_JOB_URL=https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction/-/jobs/1171
CI_NODE_TOTAL=1
CI_PAGES_DOMAIN=example.com
CI_PAGES_URL=http://test.example.com/zd134231-disjunction
CI_PIPELINE_ID=90
CI_PIPELINE_IID=90
CI_PIPELINE_SOURCE=push
CI_PIPELINE_URL=https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction/pipelines/90
CI_PROJECT_DIR=/home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction
CI_PROJECT_ID=1
CI_PROJECT_NAMESPACE=test
CI_PROJECT_NAME=zd134231-disjunction
CI_PROJECT_PATH_SLUG=test-zd134231-disjunction
CI_PROJECT_PATH=test/zd134231-disjunction
CI_PROJECT_REPOSITORY_LANGUAGES=
CI_PROJECT_URL=https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction
CI_PROJECT_VISIBILITY=internal
CI_REGISTRY_PASSWORD=[MASKED]
CI_REGISTRY_USER=gitlab-ci-token
CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction.git
CI_RUNNER_DESCRIPTION=localrunner
CI_RUNNER_EXECUTABLE_ARCH=linux/amd64
CI_RUNNER_ID=1
CI_RUNNER_REVISION=a8a019e0
CI_RUNNER_SHORT_TOKEN=J2yXYmYx
CI_RUNNER_TAGS=shell
CI_RUNNER_VERSION=12.3.0
CI_SERVER_HOST=bprescott-gitlabtest-0.do.gitlap.com
CI_SERVER_NAME=GitLab
CI_SERVER_REVISION=9dbaa740018
CI_SERVER_TLS_CA_FILE=/home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction.tmp/CI_SERVER_TLS_CA_FILE
CI_SERVER_VERSION=12.3.5-ee
CI_SERVER_VERSION_MAJOR=12
CI_SERVER_VERSION_MINOR=3
CI_SERVER_VERSION_PATCH=5
CI_SERVER=yes
CI_SHARED_ENVIRONMENT=true
CI=true
CONFIG_FILE=/etc/gitlab-runner/config.toml
FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=false
FF_USE_LEGACY_BUILDS_DIR_FOR_DOCKER=false
FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER=false
GITLAB_CI=true
GITLAB_FEATURES=
GITLAB_USER_EMAIL=bprescott@gitlab.com
GITLAB_USER_ID=1
GITLAB_USER_LOGIN=bprescott-root
GITLAB_USER_NAME=Ben Prescott (root)
gpattern1=/^abcde/
gpattern2=/^abcd/
gpattern3=/^ab./
gpattern4=/^ab/
gpattern5=/^abcde./
gpattern6=/^abcde/
gpattern7=/^./
gpattern8=/./
HISTCONTROL=ignoredups
HISTSIZE=1000
HOME=/home/gitlab-runner
HOSTNAME=bprescott-gitlabtest-0
LANG=en_US.UTF-8
LESSOPEN=||/usr/bin/lesspipe.sh %s
LOGNAME=gitlab-runner
MAIL=/var/spool/mail/gitlab-runner
OLDPWD=/home/gitlab-runner
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/home/gitlab-runner/.local/bin:/home/gitlab-runner/bin
pattern1=/^abcde/
pattern2=/^abcd/
pattern3=/^ab./
pattern4=/^ab/
pattern5=/^abcde./
pattern6=/^abcde/
pattern7=/^./
pattern8=/./
PWD=/home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction
SHELL=/bin/bash
SHLVL=2
teststring=abcde
USER=gitlab-runner
_=/usr/bin/env
XDG_RUNTIME_DIR=/run/user/992
XDG_SESSION_ID=c9078
Job succeeded
this section relates to the $ issue, which has been put in a separate issue

see: #36940 (closed)

Note, none of the dollars appear in the above output

gpattern1=/^abcde/
gpattern2=/^abcd/
gpattern3=/^ab.*/
gpattern4=/^ab/
gpattern5=/^abcde.*/
gpattern6=/^abcde/
gpattern7=/^.*/
gpattern8=/.*/
[..]
pattern1=/^abcde/
pattern2=/^abcd/
pattern3=/^ab.*/
pattern4=/^ab/
pattern5=/^abcde.*/
pattern6=/^abcde/
pattern7=/^.*/
pattern8=/.*/

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info 
# sudo gitlab-rake gitlab:env:info

System information
System:
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.3p62
Gem Version:	2.7.9
Bundler Version:1.17.3
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.22.0
Sidekiq Version:5.2.7
Go Version:	unknown


GitLab information
Version:	12.3.5-ee
Revision:	9dbaa740018
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	10.9
URL:		https://[gitlab-DO-test-machine].com
HTTP Clone URL:	https://[gitlab-DO-test-machine].com/some-group/some-project.git
SSH Clone URL:	git@[gitlab-DO-test-machine].com:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:


GitLab Shell
Version:	10.0.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
# sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 10.0.0 ? ... OK (10.0.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/1 ... yes
2/2 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.22.0 ? ... yes (2.22.0)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

I believe support for pattern matching was visited a couple of months ago:

Documentation:

Pattern matching with variables is documented with just the following line

It is possible perform pattern matching against a variable and regular expression.

Following chat with @thaoyeager I've opened #36942 (closed) around this.

An initial iteration would be to document that the feature is currently very limited; it's been working to some extent since 12.3, and is not likely to be fixed before 12.7 at the earliest. I'll explore this option in !19098 (closed)

Proposal

Force convert both sides of =~ and !~ to Regexp.

Example technical change;

diff --git a/lib/gitlab/ci/pipeline/expression/lexeme/matches.rb b/lib/gitlab/ci/pipeline/expression/lexeme/matches.rb
index 4d65b914d8d..edc61db0a2c 100644
--- a/lib/gitlab/ci/pipeline/expression/lexeme/matches.rb
+++ b/lib/gitlab/ci/pipeline/expression/lexeme/matches.rb
@@ -11,6 +11,7 @@ class Matches < Lexeme::LogicalOperator
             def evaluate(variables = {})
               text = @left.evaluate(variables)
               regexp = @right.evaluate(variables)
+              regexp = Gitlab::UntrustedRegexp::RubySyntax.fabricate(regexp) || regexp
               return false unless regexp

               regexp.scan(text.to_s).present?
diff --git a/lib/gitlab/ci/pipeline/expression/lexeme/not_matches.rb b/lib/gitlab/ci/pipeline/expression/lexeme/not_matches.rb
index 29c5aa5d753..00338886310 100644
--- a/lib/gitlab/ci/pipeline/expression/lexeme/not_matches.rb
+++ b/lib/gitlab/ci/pipeline/expression/lexeme/not_matches.rb
@@ -11,6 +11,7 @@ class NotMatches < Lexeme::LogicalOperator
             def evaluate(variables = {})
               text = @left.evaluate(variables)
               regexp = @right.evaluate(variables)
+              regexp = Gitlab::UntrustedRegexp::RubySyntax.fabricate(regexp) || regexp
               return true unless regexp

               regexp.scan(text.to_s).empty?
Edited by Mark Nuzzo