Skip to content
GitLab
Next
    • GitLab: the DevOps platform
    • Explore GitLab
    • Install GitLab
    • How GitLab compares
    • Get started
    • GitLab docs
    • GitLab Learn
  • Pricing
  • Talk to an expert
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
    Projects Groups Topics Snippets
  • Register
  • Sign in
  • gitlab-runner gitlab-runner
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
    • Locked files
  • Issues 3,456
    • Issues 3,456
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 106
    • Merge requests 106
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Artifacts
    • Schedules
    • Test cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • gitlab-runnergitlab-runner
  • Issues
  • #28632
Closed
Open
Issue created Nov 03, 2021 by Üllar Seerme@yllar.seerme

Piping into 'tail' first seems to cause out-of-order execution

Summary

Piping into tail first seems to cause out-of-order execution when working with multiple piped operations.

Steps to reproduce

The following CI YAML has everything to reproduce this issue.

.gitlab-ci.yml
variables:
  MESSAGE: |
    git@example.com:group/project
    subscription: sub1
    location: loc1

.shared-scripts:
  set-vars: &set-vars
    - >
      SUBSCRIPTION=$(echo "$MESSAGE" | tail -n 2 | head -n 1 | cut -d " " -f 2);
      LOCATION=$(echo "$MESSAGE" | tail -n 1 | cut -d " " -f 2);

anchored-job:
  before_script:
    - *set-vars
  script:
    - echo "'${SUBSCRIPTION}'"  # does not work
    - echo "'${LOCATION}'"  # does not work

regular-job-tail-first:
  before_script:
    - >
      SUBSCRIPTION=$(echo "$MESSAGE" | tail -n 2 | head -n 1 | cut -d " " -f 2);
      LOCATION=$(echo "$MESSAGE" | tail -n 1 | cut -d " " -f 2);
  script:
    - echo "'${SUBSCRIPTION}'"  # does not work
    - echo "'${LOCATION}'"  # does not work

regular-job-head-first:
  before_script:
    - >
      SUBSCRIPTION=$(echo "$MESSAGE" | head -n 2 | tail -n 1 | cut -d " " -f 2);
      LOCATION=$(echo "$MESSAGE" | tail -n 1 | cut -d " " -f 2);
  script:
    - echo "'${SUBSCRIPTION}'"  # works
    - echo "'${LOCATION}'"  # does not work

regular-job-more-pipes:
  before_script:
    - PROJECT=$(echo "$MESSAGE" | head -n 1 | cut -d ":" -f 2 | rev | cut -d "/" -f 1 | rev)
  script:
    - echo "'${PROJECT}'"  # works

Actual behavior

What is happening is that instead of every piped operation being applied in the order they are specified, some are being done in an order that is not expected. When looking at the debug output the following is apparent:

regular-job-tail-first
+++ echo 'git@example.com:group/project
subscription: sub1
location: loc1
'
+++ head -n 1
+++ tail -n 2
+++ cut -d ' ' -f 2
++ SUBSCRIPTION=loc1
+++ echo 'git@example.com:group/project
subscription: sub1
location: loc1
'
+++ cut -d ' ' -f 2
+++ tail -n 1
++ LOCATION=

Expected behavior

The order of execution should be deterministic. The SUBSCRIPTION and LOCATION command substitutions should yield sub1 and loc1, respectively.

Relevant logs and/or screenshots

anchored-job
$ SUBSCRIPTION=$(echo "$MESSAGE" | tail -n 2 | head -n 1 | cut -d " " -f 2); LOCATION=$(echo "$MESSAGE" | tail -n 1 | cut -d " " -f 2);
$ echo "'${SUBSCRIPTION}'"
'loc1'
$ echo "'${LOCATION}'"
''
regular-job-head-first
$ SUBSCRIPTION=$(echo "$MESSAGE" | head -n 2 | tail -n 1 | cut -d " " -f 2); LOCATION=$(echo "$MESSAGE" | tail -n 1 | cut -d " " -f 2);
$ echo "'${SUBSCRIPTION}'"
'sub1'
$ echo "'${LOCATION}'"
''
regular-job-tail-first
$ SUBSCRIPTION=$(echo "$MESSAGE" | tail -n 2 | head -n 1 | cut -d " " -f 2); LOCATION=$(echo "$MESSAGE" | tail -n 1 | cut -d " " -f 2);
$ echo "'${SUBSCRIPTION}'"
'loc1'
$ echo "'${LOCATION}'"
''
regular-job-more-pipes
$ PROJECT=$(echo "$MESSAGE" | head -n 1 | cut -d ":" -f 2 | rev | cut -d "/" -f 1 | rev)
$ echo "'${PROJECT}'"
'project'

Environment description

Using shared Runners for reporting this particular issue, but discovered this issue whilst using Runners on GCP.

Used GitLab Runner version

Shared Runner:

Running with gitlab-runner 14.4.0-rc1 (bc99a056)
  on docker-auto-scale 72989761

Private Runner:

Running with gitlab-runner 14.3.2 (e0218c92)
Edited Nov 03, 2021 by Üllar Seerme
Assignee
Assign to
Time tracking