Dynamic child pipelines fail in Windows runners because of line ending conversion
Summary
Trying to create dynamic child pipelines following the JSONNET example that is linked to in the offical documentation. I am using a Powershell script to generate the YAML file, and Powershell in the runner. The child pipeline consistently fails with Included file ... does not have valid YAML syntax!
Steps to reproduce
Use a generated artifact in a trigger job using a Windows 10 runner with a Powershell shell. Gitlab version is 13.3.6-ee (d7bc82f4).
Example Project
Simplified project provided here that shows a dynamically generated child pipeline that is completely valid, but fails due to invalid YAML. Project includes a parallel job that reads and echos the generated YAML file to confirm it is present and valid.
What is the current bug behavior?
Generated YAML file is
stages:
- sanity
.configuration:
variables:
TARGET: TLS4_Q4Q4N4V17
PLATFORM: G2.0
ARCH: x86
tags:
- gen2
- TLS
sanity:
stage: sanity
script:
- dir
extends:
- .configuration
This file passes lint, but consistently fails the pipeline. No matter what I change, the child pipeline will not run and says the syntax is invalid. Every failure is checked in lint and passes
I note the following two issues and taken the given precautions:
- #227802 (closed): Removed all rules from dynamically generated child file as well as parent file
- #212373 (closed): In testing, I had a parallel job that used the generation job and ensured the YAML file was generated and present. This is commented out in the example. It fails whether there is a parallel job or not.
I've confirmed that the generated YAML files works if the child pipeline. Taking the resulting artifact and renaming the file to generated.gitlab-ci.yml
results in a passing child pipeline using the following job definition:
run-child:
trigger:
include:
- local: generated.gitlab-ci.yml
I've confirmed that the naming strategy TLS4_Q4Q4N4V17-gitlab-ci.yml
is not the issue, as the pipeline fails when generated with the naming strategy generated.gitlab-ci.yml
, exactly the same name as passes above.
I've confirmed that providing generated.gitlab-ci.yml
in source control results in a passing child pipeline, as the pipeline appears to read the indexed file rather than the artifact. This suggests the artifact is not present, but I've also confirmed that a non-present artifact results in the error message Path 'generated-config.yml' does not exist inside the 'generate' artifacts archive!
. Thus, this further confirms the generated file is present.
What is the expected correct behavior?
Child pipeline should run, or specific error should propigate if invalid YAML syntax is a red herring.
Relevant logs and/or screenshots
Error is prior to runner picking up the job, no relevant logs found.
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 9.13 Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.9 Git Version: 2.28.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.3.6-ee Revision: d7bc82f4b06 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.7 URL: http://gitlab.anpc.com HTTP Clone URL: http://gitlab.anpc.com/some-group/some-project.git SSH Clone URL: git@gitlab.anpc.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 13.6.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.6.0 ? ... OK (13.6.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 52 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 44/8 ... yes 44/9 ... yes 44/10 ... yes 44/11 ... yes 44/12 ... yes 44/13 ... yes 44/14 ... yes 44/15 ... yes 44/16 ... yes 44/17 ... yes 44/18 ... yes 43/21 ... yes 39/22 ... yes 39/23 ... yes 39/61 ... yes 40/64 ... yes 39/70 ... yes 6/71 ... yes 6/73 ... yes 53/76 ... yes 43/77 ... yes 49/79 ... yes 49/80 ... yes 39/81 ... yes 56/83 ... yes 49/84 ... yes 55/85 ... yes 53/88 ... yes 40/92 ... yes 55/93 ... yes 57/94 ... yes 57/96 ... yes 57/97 ... yes 57/98 ... yes 57/99 ... yes 57/102 ... yes 57/104 ... yes 64/105 ... yes 55/106 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.24.0 ? ... yes (2.28.0) Git user has default SSH configuration? ... yes Active users: ... 18 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... no Try fixing it: Please migrate all projects to hashed storage as legacy storage is deprecated in 13.0 and support will be removed in 14.0. For more information see: doc/administration/repository_storage_types.md Elasticsearch version 6.x - 7.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes --> Workaround
After considerable debugging, I've determined that a Linux Gitlab server with Windows 10 runners mixes line endings. With Windows runners, it is imperative that you convert line endings to Linux because the context of the dynamic pipeline is Linux while it's "grabbing" the pipeline, and only becomes a Windows context after the pipeline is successfully consumed and linted. A workaround for Windows runners is below. Generated files must be converted from Windows to Unix newlines.
I feel like this should be done at the framework level--not left to pipeline writers. At the very least, it should be documented in the pipeline documentation that Windows runners require line ending conversion.
File causing the behavior detailed above
stages: - build - testgenerate: stage: build script: - .\pipeline.ps1 -t TLS4_Q4Q4N4V17 > generated.yml artifacts: paths: - generated-config.yml tags: - TLS
run-child: stage: test needs: - job: generate artifacts: true trigger: include: - artifact: generated-config.yml job: generate
File allowing dynamic pipelines with Windows Runners
stages: - build - testgenerate: stage: build script: - .\pipeline.ps1 -t TLS4_Q4Q4N4V17 > generated.yml - Get-Content generated.yml | Out-File -encoding ascii -filepath generated-config.yml artifacts: paths: - generated-config.yml tags: - TLS
run-child: stage: test needs: - job: generate artifacts: true trigger: include: - artifact: generated-config.yml job: generate