Pipelines - variables containing $ are exported by the runner with $ removed
Summary
If a pipeline variable contains a $
(such as part of a regular expression) it is intact in the variables hash, but when that variable is exported to the runner (and in the runner debug output) the $
is missing
Originally part of #35438 (closed). Analysis from comments:
Regarding the missing
$
, it does not cause any problems in the regex pattern matching because when I debugged thevariables
hash, the values are still intact with the$
in them. So maybe this can be addressed on a separate issue.
Steps to reproduce
While reproducing an issue relating to regular expressions in variables, I found it originally using CI_COMMIT_DESCRIPTION, and this is what I captured:
Author: Ben Prescott <bprescott@example.com>
Date: Fri Oct 25 15:16:14 2019 +0100
ci: add a description
/^mo$/
++ export 'CI_COMMIT_DESCRIPTION=/^mo/'
++ CI_COMMIT_DESCRIPTION='/^mo/'
I've got an example project linked; if you look at the dumped variables in the output/screenshot section below, all of the variables containing $
have had it removed.
Example Project
Note: I'm testing on a standalone install, and my project is configured to expect the CI yaml definition in ci/definition.yml
This should show as commit edccc5dd
which matches screen shots etc. below.
What is the current bug behavior?
$
doesn't appear in the debug output and exported to the runner.
What is the expected correct behavior?
Bearing in mind this is the only debug mechanism .. strings make it through verbatim.
There might be times when customers don't expect it to be missing. For example .. commit messages, or any other variable that could contain any valid character.
Shell does support dollars in variables, even though it has special meaning in that context:
~$ export foo='1234$'
~$ echo $foo
1234$
Relevant logs and/or screenshots
Expand for output of environment dump
Running with gitlab-runner 12.3.0 (a8a019e0) on localrunner J2yXYmYx Using Shell executor... Running on bprescott-gitlabtest-0... Fetching changes with git depth set to 50... Reinitialized existing Git repository in /home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction/.git/ From https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction + 0ffbfd0...edccc5d 20191101-35438 -> origin/20191101-35438 (forced update) Checking out edccc5dd as 20191101-35438... Skipping Git submodules setup $ /usr/bin/env | /usr/bin/sortCI_API_V4_URL=https://bprescott-gitlabtest-0.do.gitlap.com/api/v4 CI_BUILD_BEFORE_SHA=0ffbfd0089d1d32f7be2ab337310fc89a023e1b8 CI_BUILD_ID=1171 CI_BUILD_NAME=envdump CI_BUILD_REF=edccc5dd95833c908bcac8a3ea3e187a3bb7e479 CI_BUILD_REF_NAME=20191101-35438 CI_BUILD_REF_SLUG=20191101-35438 CI_BUILDS_DIR=/home/gitlab-runner/builds CI_BUILD_STAGE=debug CI_BUILD_TOKEN=[MASKED] CI_COMMIT_BEFORE_SHA=0ffbfd0089d1d32f7be2ab337310fc89a023e1b8 CI_COMMIT_DESCRIPTION= CI_COMMIT_MESSAGE=ci: compare hard coded patterns with variables - for gitlab issue 35438 CI_COMMIT_REF_NAME=20191101-35438 CI_COMMIT_REF_PROTECTED=false CI_COMMIT_REF_SLUG=20191101-35438 CI_COMMIT_SHA=edccc5dd95833c908bcac8a3ea3e187a3bb7e479 CI_COMMIT_SHORT_SHA=edccc5dd CI_COMMIT_TITLE=ci: compare hard coded patterns with variables - for gitlab issue 35438 CI_CONCURRENT_ID=0 CI_CONCURRENT_PROJECT_ID=0 CI_CONFIG_PATH=ci/definition.yml CI_JOB_ID=1171 CI_JOB_NAME=envdump CI_JOB_STAGE=debug CI_JOB_TOKEN=[MASKED] CI_JOB_URL=https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction/-/jobs/1171 CI_NODE_TOTAL=1 CI_PAGES_DOMAIN=example.com CI_PAGES_URL=http://test.example.com/zd134231-disjunction CI_PIPELINE_ID=90 CI_PIPELINE_IID=90 CI_PIPELINE_SOURCE=push CI_PIPELINE_URL=https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction/pipelines/90 CI_PROJECT_DIR=/home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction CI_PROJECT_ID=1 CI_PROJECT_NAMESPACE=test CI_PROJECT_NAME=zd134231-disjunction CI_PROJECT_PATH_SLUG=test-zd134231-disjunction CI_PROJECT_PATH=test/zd134231-disjunction CI_PROJECT_REPOSITORY_LANGUAGES= CI_PROJECT_URL=https://bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction CI_PROJECT_VISIBILITY=internal CI_REGISTRY_PASSWORD=[MASKED] CI_REGISTRY_USER=gitlab-ci-token CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@bprescott-gitlabtest-0.do.gitlap.com/test/zd134231-disjunction.git CI_RUNNER_DESCRIPTION=localrunner CI_RUNNER_EXECUTABLE_ARCH=linux/amd64 CI_RUNNER_ID=1 CI_RUNNER_REVISION=a8a019e0 CI_RUNNER_SHORT_TOKEN=J2yXYmYx CI_RUNNER_TAGS=shell CI_RUNNER_VERSION=12.3.0 CI_SERVER_HOST=bprescott-gitlabtest-0.do.gitlap.com CI_SERVER_NAME=GitLab CI_SERVER_REVISION=9dbaa740018 CI_SERVER_TLS_CA_FILE=/home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction.tmp/CI_SERVER_TLS_CA_FILE CI_SERVER_VERSION=12.3.5-ee CI_SERVER_VERSION_MAJOR=12 CI_SERVER_VERSION_MINOR=3 CI_SERVER_VERSION_PATCH=5 CI_SERVER=yes CI_SHARED_ENVIRONMENT=true CI=true CONFIG_FILE=/etc/gitlab-runner/config.toml FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=false FF_USE_LEGACY_BUILDS_DIR_FOR_DOCKER=false FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER=false GITLAB_CI=true GITLAB_FEATURES= GITLAB_USER_EMAIL=bprescott@gitlab.com GITLAB_USER_ID=1 GITLAB_USER_LOGIN=bprescott-root GITLAB_USER_NAME=Ben Prescott (root) gpattern1=/^abcde/ gpattern2=/^abcd/ gpattern3=/^ab./ gpattern4=/^ab/ gpattern5=/^abcde./ gpattern6=/^abcde/ gpattern7=/^./ gpattern8=/./ HISTCONTROL=ignoredups HISTSIZE=1000 HOME=/home/gitlab-runner HOSTNAME=bprescott-gitlabtest-0 LANG=en_US.UTF-8 LESSOPEN=||/usr/bin/lesspipe.sh %s LOGNAME=gitlab-runner MAIL=/var/spool/mail/gitlab-runner OLDPWD=/home/gitlab-runner PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/home/gitlab-runner/.local/bin:/home/gitlab-runner/bin pattern1=/^abcde/ pattern2=/^abcd/ pattern3=/^ab./ pattern4=/^ab/ pattern5=/^abcde./ pattern6=/^abcde/ pattern7=/^./ pattern8=/./ PWD=/home/gitlab-runner/builds/J2yXYmYx/0/test/zd134231-disjunction SHELL=/bin/bash SHLVL=2 teststring=abcde USER=gitlab-runner _=/usr/bin/env XDG_RUNTIME_DIR=/run/user/992 XDG_SESSION_ID=c9078 Job succeeded
Note, none of the dollars appear in the above output
gpattern1=/^abcde/
gpattern2=/^abcd/
gpattern3=/^ab.*/
gpattern4=/^ab/
gpattern5=/^abcde.*/
gpattern6=/^abcde/
gpattern7=/^.*/
gpattern8=/.*/
[..]
pattern1=/^abcde/
pattern2=/^abcd/
pattern3=/^ab.*/
pattern4=/^ab/
pattern5=/^abcde.*/
pattern6=/^abcde/
pattern7=/^.*/
pattern8=/.*/
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
# sudo gitlab-rake gitlab:env:infoSystem information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: unknown
GitLab information Version: 12.3.5-ee Revision: 9dbaa740018 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://[gitlab-DO-test-machine].com HTTP Clone URL: https://[gitlab-DO-test-machine].com/some-group/some-project.git SSH Clone URL: git@[gitlab-DO-test-machine].com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 10.0.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
# sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 10.0.0 ? ... OK (10.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 2/2 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.22.0) Git user has default SSH configuration? ... yes Active users: ... 1 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
It would be a valid iteration to document this as a limitation ahead of fixing it. I will explore doing this in !19098 (closed)