Piping into 'tail' first seems to cause out-of-order execution
Summary
Piping into tail
first seems to cause out-of-order execution when working with multiple piped operations.
Steps to reproduce
The following CI YAML has everything to reproduce this issue.
.gitlab-ci.yml
variables:
MESSAGE: |
git@example.com:group/project
subscription: sub1
location: loc1
.shared-scripts:
set-vars: &set-vars
- >
SUBSCRIPTION=$(echo "$MESSAGE" | tail -n 2 | head -n 1 | cut -d " " -f 2);
LOCATION=$(echo "$MESSAGE" | tail -n 1 | cut -d " " -f 2);
anchored-job:
before_script:
- *set-vars
script:
- echo "'${SUBSCRIPTION}'" # does not work
- echo "'${LOCATION}'" # does not work
regular-job-tail-first:
before_script:
- >
SUBSCRIPTION=$(echo "$MESSAGE" | tail -n 2 | head -n 1 | cut -d " " -f 2);
LOCATION=$(echo "$MESSAGE" | tail -n 1 | cut -d " " -f 2);
script:
- echo "'${SUBSCRIPTION}'" # does not work
- echo "'${LOCATION}'" # does not work
regular-job-head-first:
before_script:
- >
SUBSCRIPTION=$(echo "$MESSAGE" | head -n 2 | tail -n 1 | cut -d " " -f 2);
LOCATION=$(echo "$MESSAGE" | tail -n 1 | cut -d " " -f 2);
script:
- echo "'${SUBSCRIPTION}'" # works
- echo "'${LOCATION}'" # does not work
regular-job-more-pipes:
before_script:
- PROJECT=$(echo "$MESSAGE" | head -n 1 | cut -d ":" -f 2 | rev | cut -d "/" -f 1 | rev)
script:
- echo "'${PROJECT}'" # works
Actual behavior
What is happening is that instead of every piped operation being applied in the order they are specified, some are being done in an order that is not expected. When looking at the debug output the following is apparent:
regular-job-tail-first
+++ echo 'git@example.com:group/project
subscription: sub1
location: loc1
'
+++ head -n 1
+++ tail -n 2
+++ cut -d ' ' -f 2
++ SUBSCRIPTION=loc1
+++ echo 'git@example.com:group/project
subscription: sub1
location: loc1
'
+++ cut -d ' ' -f 2
+++ tail -n 1
++ LOCATION=
Expected behavior
The order of execution should be deterministic. The SUBSCRIPTION
and LOCATION
command substitutions should yield sub1
and loc1
, respectively.
Relevant logs and/or screenshots
anchored-job
$ SUBSCRIPTION=$(echo "$MESSAGE" | tail -n 2 | head -n 1 | cut -d " " -f 2); LOCATION=$(echo "$MESSAGE" | tail -n 1 | cut -d " " -f 2);
$ echo "'${SUBSCRIPTION}'"
'loc1'
$ echo "'${LOCATION}'"
''
regular-job-head-first
$ SUBSCRIPTION=$(echo "$MESSAGE" | head -n 2 | tail -n 1 | cut -d " " -f 2); LOCATION=$(echo "$MESSAGE" | tail -n 1 | cut -d " " -f 2);
$ echo "'${SUBSCRIPTION}'"
'sub1'
$ echo "'${LOCATION}'"
''
regular-job-tail-first
$ SUBSCRIPTION=$(echo "$MESSAGE" | tail -n 2 | head -n 1 | cut -d " " -f 2); LOCATION=$(echo "$MESSAGE" | tail -n 1 | cut -d " " -f 2);
$ echo "'${SUBSCRIPTION}'"
'loc1'
$ echo "'${LOCATION}'"
''
regular-job-more-pipes
$ PROJECT=$(echo "$MESSAGE" | head -n 1 | cut -d ":" -f 2 | rev | cut -d "/" -f 1 | rev)
$ echo "'${PROJECT}'"
'project'
Environment description
Using shared Runners for reporting this particular issue, but discovered this issue whilst using Runners on GCP.
Used GitLab Runner version
Shared Runner:
Running with gitlab-runner 14.4.0-rc1 (bc99a056)
on docker-auto-scale 72989761
Private Runner:
Running with gitlab-runner 14.3.2 (e0218c92)