Backend: Regexp CI/CD variables are evaluated as strings
Summary
When you have a Regexp variable, you cannot use it as a Regexp in rules
/only
conditions.
For example;
variables:
teststring: 'abcde'
pattern: '/^ab.*/'
test1:
script: exit 0
rules:
- if: '$teststring =~ $pattern'
test2:
script: exit 0
rules:
- if: '$teststring =~ /^ab.*/'
- The
test1
job is not created because the backend makes string comparison between"abcde"
and"/'^ab.*/"
. - The
test2
job is created because the backend makes regexp comparison between"abcde"
and/'^ab.*/
.
Customer background
-
Customer ticket (internal) was around the fact that any pattern definition in a variable containing
.*
could not be made to work, having tried several protection strategies to define and / or use the variable- I might have found a further misbehaviour, which is that while
/^ab/
matchesabcde
when applied direct, it doesn't work when matching via a variable. My further test case of/^ab.*/
probably doesn't work because/^ab/
doesn't.
- I might have found a further misbehaviour, which is that while
-
(now in #36940 (closed)) Jobs do get created when the regex (in a variable) contains
$
but when that variable is exported to the runner (and in the runner debug output) the$
is missing
edit 2019-11-21 - title changed, and content largely hidden for the issue relating to $
, now: #36940 (closed)
Steps to reproduce
The example project contains a full CI configuration; this is a summary.
These are the test patterns and test string.
variables:
teststring: 'abcde'
pattern1: '/^abcde$/'
pattern2: '/^abcd$/'
pattern3: '/^ab.*/'
pattern4: '/^ab/'
pattern5: '/^abcde.*/'
pattern6: '/^abcde/'
pattern7: '/^.*$/'
pattern8: '/.*/'
this section relates to the $ issue, which has been put in a separate issue
see: #36940 (closed)
In respect of $
one of my example jobs dumps the shell environment. I forgot to enable debugging. I found it originally using CI_COMMIT_DESCRIPTION, and this is what I captured:
Author: Ben Prescott <bprescott@example.com>
Date: Fri Oct 25 15:16:14 2019 +0100
ci: add a description
/^mo$/
++ export 'CI_COMMIT_DESCRIPTION=/^mo/'
++ CI_COMMIT_DESCRIPTION='/^mo/'
Reverting to the example I made for this ticket, if you look at the dumped variables in the output/screenshot section below, all of the variables containing $
have had it removed.
Moving on from $
to the pattern matching issue ..
Defining job the following way, using only
, seven jobs get created. pattern2 fails as expected.
Job definition where pattern match is applied direct
direct5: stage: pmdirect tags: - shell script: - /usr/bin/true only: variables: - $teststring =~ /^abcde.*/
Via variables, only three patterns work - 1, 5, and 6. I've tried it with both only
and rules
Job definition where pattern match is applied via variable
variable1: stage: pmvariable tags: - shell script: - /usr/bin/true only: variables: - $teststring =~ $pattern1 # varrule1: stage: pmvarrule tags: - shell script: - /usr/bin/true rules: - if: '$teststring =~ $pattern1' when: on_success
I've also created a set of variables in the GUI, to explore whether reading the regexes from the yaml is where it goes wrong.
Same result.
Example Project
Note: I'm testing on a standalone install, and my project is configured to expect the CI yaml definition in ci/definition.yml
This should show as commit edccc5dd
which matches screen shots etc. below.
What is the current bug behavior?
-
Jobs that get created when a pattern match rule works explicitly fail to get create when the pattern is stored in a variable instead. Patterns including
.*
don't work, and this additional/^ab/
case which I only just found, and am not sure what the general case is. -
(now in #36940 (closed))
$
doesn't appear in the debug output and exported to the runner. Issue was noticed investigating the other issue.
What is the expected correct behavior?
-
The same jobs that get created when pattern matched explicitly should get created when the patterns are in variables.
-
(now in #36940 (closed)) Bearing in mind this is the only debug mechanism .. strings make it through verbatim. It may well be the case this simply isn't possible, because
$
has a specific meaning in shell! There might be times when customers don't expect it to be missing. For example .. commit messages, or any other variable that could contain any valid character.
Relevant logs and/or screenshots
Output from example project.
Expand for output of environment dump
Running with gitlab-runner 12.3.0 (a8a019e0) on localrunner J2yXYmYx Using Shell executor... Running on bprescott-gitlabtest-0... Fetching changes with git depth set to 50... Reinitialized existing Git repository in /home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction/.git/ From https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction + 0ffbfd0...edccc5d 20191101-35438 -> origin/20191101-35438 (forced update) Checking out edccc5dd as 20191101-35438... Skipping Git submodules setup $ /usr/bin/env | /usr/bin/sortCI_API_V4_URL=https://bprescott-gitlabtest-0.do.gitlap.com/api/v4 CI_BUILD_BEFORE_SHA=0ffbfd0089d1d32f7be2ab337310fc89a023e1b8 CI_BUILD_ID=1171 CI_BUILD_NAME=envdump CI_BUILD_REF=edccc5dd95833c908bcac8a3ea3e187a3bb7e479 CI_BUILD_REF_NAME=20191101-35438 CI_BUILD_REF_SLUG=20191101-35438 CI_BUILDS_DIR=/home/gitlab-runner/builds CI_BUILD_STAGE=debug CI_BUILD_TOKEN=[MASKED] CI_COMMIT_BEFORE_SHA=0ffbfd0089d1d32f7be2ab337310fc89a023e1b8 CI_COMMIT_DESCRIPTION= CI_COMMIT_MESSAGE=ci: compare hard coded patterns with variables - for gitlab issue 35438 CI_COMMIT_REF_NAME=20191101-35438 CI_COMMIT_REF_PROTECTED=false CI_COMMIT_REF_SLUG=20191101-35438 CI_COMMIT_SHA=edccc5dd95833c908bcac8a3ea3e187a3bb7e479 CI_COMMIT_SHORT_SHA=edccc5dd CI_COMMIT_TITLE=ci: compare hard coded patterns with variables - for gitlab issue 35438 CI_CONCURRENT_ID=0 CI_CONCURRENT_PROJECT_ID=0 CI_CONFIG_PATH=ci/definition.yml CI_JOB_ID=1171 CI_JOB_NAME=envdump CI_JOB_STAGE=debug CI_JOB_TOKEN=[MASKED] CI_JOB_URL=https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction/-/jobs/1171 CI_NODE_TOTAL=1 CI_PAGES_DOMAIN=example.com CI_PAGES_URL=http://test.example.com/zd134231-disjunction CI_PIPELINE_ID=90 CI_PIPELINE_IID=90 CI_PIPELINE_SOURCE=push CI_PIPELINE_URL=https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction/pipelines/90 CI_PROJECT_DIR=/home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction CI_PROJECT_ID=1 CI_PROJECT_NAMESPACE=test CI_PROJECT_NAME=zd134231-disjunction CI_PROJECT_PATH_SLUG=test-zd134231-disjunction CI_PROJECT_PATH=test/zd134231-disjunction CI_PROJECT_REPOSITORY_LANGUAGES= CI_PROJECT_URL=https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction CI_PROJECT_VISIBILITY=internal CI_REGISTRY_PASSWORD=[MASKED] CI_REGISTRY_USER=gitlab-ci-token CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction.git CI_RUNNER_DESCRIPTION=localrunner CI_RUNNER_EXECUTABLE_ARCH=linux/amd64 CI_RUNNER_ID=1 CI_RUNNER_REVISION=a8a019e0 CI_RUNNER_SHORT_TOKEN=J2yXYmYx CI_RUNNER_TAGS=shell CI_RUNNER_VERSION=12.3.0 CI_SERVER_HOST=bprescott-gitlabtest-0.do.gitlap.com CI_SERVER_NAME=GitLab CI_SERVER_REVISION=9dbaa740018 CI_SERVER_TLS_CA_FILE=/home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction.tmp/CI_SERVER_TLS_CA_FILE CI_SERVER_VERSION=12.3.5-ee CI_SERVER_VERSION_MAJOR=12 CI_SERVER_VERSION_MINOR=3 CI_SERVER_VERSION_PATCH=5 CI_SERVER=yes CI_SHARED_ENVIRONMENT=true CI=true CONFIG_FILE=/etc/gitlab-runner/config.toml FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=false FF_USE_LEGACY_BUILDS_DIR_FOR_DOCKER=false FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER=false GITLAB_CI=true GITLAB_FEATURES= GITLAB_USER_EMAIL=bprescott@gitlab.com GITLAB_USER_ID=1 GITLAB_USER_LOGIN=bprescott-root GITLAB_USER_NAME=Ben Prescott (root) gpattern1=/^abcde/ gpattern2=/^abcd/ gpattern3=/^ab./ gpattern4=/^ab/ gpattern5=/^abcde./ gpattern6=/^abcde/ gpattern7=/^./ gpattern8=/./ HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/gitlab-runner HOSTNAME=bprescott-gitlabtest-0 LANG=en_US.UTF-8 LESSOPEN=||/usr/bin/lesspipe.sh %s LOGNAME=gitlab-runner MAIL=/var/spool/mail/gitlab-runner OLDPWD=/home/gitlab-runner PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/home/gitlab-runner/.local/bin:/home/gitlab-runner/bin pattern1=/^abcde/ pattern2=/^abcd/ pattern3=/^ab./ pattern4=/^ab/ pattern5=/^abcde./ pattern6=/^abcde/ pattern7=/^./ pattern8=/./ PWD=/home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction SHELL=/bin/bash SHLVL=2 teststring=abcde USER=gitlab-runner _=/usr/bin/env XDG_RUNTIME_DIR=/run/user/992 XDG_SESSION_ID=c9078 Job succeeded
this section relates to the $ issue, which has been put in a separate issue
see: #36940 (closed)
Note, none of the dollars appear in the above output
gpattern1=/^abcde/
gpattern2=/^abcd/
gpattern3=/^ab.*/
gpattern4=/^ab/
gpattern5=/^abcde.*/
gpattern6=/^abcde/
gpattern7=/^.*/
gpattern8=/.*/
[..]
pattern1=/^abcde/
pattern2=/^abcd/
pattern3=/^ab.*/
pattern4=/^ab/
pattern5=/^abcde.*/
pattern6=/^abcde/
pattern7=/^.*/
pattern8=/.*/
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
# sudo gitlab-rake gitlab:env:infoSystem information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: unknown
GitLab information Version: 12.3.5-ee Revision: 9dbaa740018 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://[gitlab-DO-test-machine].com HTTP Clone URL: https://[gitlab-DO-test-machine].com/some-group/some-project.git SSH Clone URL: git@[gitlab-DO-test-machine].com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 10.0.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
# sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 10.0.0 ? ... OK (10.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 2/2 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.22.0) Git user has default SSH configuration? ... yes Active users: ... 1 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
I believe support for pattern matching was visited a couple of months ago:
Documentation:
Pattern matching with variables is documented with just the following line
It is possible perform pattern matching against a variable and regular expression.
Following chat with @thaoyeager I've opened #36942 (closed) around this.
An initial iteration would be to document that the feature is currently very limited; it's been working to some extent since 12.3, and is not likely to be fixed before 12.7 at the earliest. I'll explore this option in !19098 (closed)
Proposal
Force convert both sides of =~
and !~
to Regexp.
Example technical change;
diff --git a/lib/gitlab/ci/pipeline/expression/lexeme/matches.rb b/lib/gitlab/ci/pipeline/expression/lexeme/matches.rb
index 4d65b914d8d..edc61db0a2c 100644
--- a/lib/gitlab/ci/pipeline/expression/lexeme/matches.rb
+++ b/lib/gitlab/ci/pipeline/expression/lexeme/matches.rb
@@ -11,6 +11,7 @@ class Matches < Lexeme::LogicalOperator
def evaluate(variables = {})
text = @left.evaluate(variables)
regexp = @right.evaluate(variables)
+ regexp = Gitlab::UntrustedRegexp::RubySyntax.fabricate(regexp) || regexp
return false unless regexp
regexp.scan(text.to_s).present?
diff --git a/lib/gitlab/ci/pipeline/expression/lexeme/not_matches.rb b/lib/gitlab/ci/pipeline/expression/lexeme/not_matches.rb
index 29c5aa5d753..00338886310 100644
--- a/lib/gitlab/ci/pipeline/expression/lexeme/not_matches.rb
+++ b/lib/gitlab/ci/pipeline/expression/lexeme/not_matches.rb
@@ -11,6 +11,7 @@ class NotMatches < Lexeme::LogicalOperator
def evaluate(variables = {})
text = @left.evaluate(variables)
regexp = @right.evaluate(variables)
+ regexp = Gitlab::UntrustedRegexp::RubySyntax.fabricate(regexp) || regexp
return true unless regexp
regexp.scan(text.to_s).empty?