Triggering multi-project pipelines fail at 2nd level with "downstream pieline trigger definition invalid"

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Hi there, firstoff: Thanks for your hard work and I hope our issue isn't caused by me overlooking something in the documentation.

Summary

Triggering multi-project 2nd level downstream pipelines fails with "downstream pipeline trigger definition invalid".

I also found this forum thread that seems to refer to the same issue ( after they disabled paralell )

https://forum.gitlab.com/t/downstream-pipeline-nesting-broken/94229

What is the expected correct behavior?

If I understand correctly, according to https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html [2024.02.19] multi project pipelines should not be affected by nesting limits. So I would expect the pipelines to trigger without "definition invalid" errors. Definitions are obviously valid as triggering works as long they're no more than 2 lvls downstream. [click to expand for screenshot ]

Versions/Facts

Selfhosted, image from docker.io [Click to expand versions]

FROM docker.io/gitlab/gitlab-ce:latest

GitLabv16.8.2 GitLab Shell14.33.0 GitLab Workhorsev16.8.2 GitLab APIv4 GitLab Pages16.8.2 GitLab KASv16.9.0-rc1 Ruby3.1.4p223 Rails7.0.8 PostgreSQL (main)13.13 PostgreSQL (ci)13.13 Redis7.0.15

Steps to reproduce

• Start pipeline with multi project triggers
• triggering fails at 2nd level of "Downstream"

Example Project

This public multi project pipeline:

https://git.conesphere.cloud/conesphere/podman/hub/debian.stable/-/pipelines/28011

What is the current bug behavior?

If we look at the path: ✓ "debian.stable" -> ✓ "apache" -> X "php_fpm" -> X "nextcloud"

Triggering php_fpm fails with "downstream pipeline trigger definition invalid".

If we start our pipeline at "apache":

https://git.conesphere.cloud/conesphere/podman/hub/apache/-/pipelines/27977

                      ✓ "apache"  ->  ✓ "php_fpm"  -> X "nextcloud"

If we start at "php_fpm": https://git.conesphere.cloud/conesphere/podman/hub/php_fpm/-/pipelines/27962 ✓ "php_fpm" -> ✓ "nextcloud"

Relevant logs, links and/or screenshots

Links to referenced project pipeline overview pages:

• https://git.conesphere.cloud/conesphere/podman/hub/debian.stable/-/pipelines
• https://git.conesphere.cloud/conesphere/podman/hub/apache/-/pipelines
• https://git.conesphere.cloud/conesphere/podman/hub/php_fpm/-/pipelines

--> The triggering user "csph_pipeline_user" is allowed to trigger all referenced pipelines

Trigger definitions are taken from https://git.conesphere.cloud/conesphere/misc/conesphere.gitlab-ci.yml-snippets/-/blob/fca3ee6be5427f022264413916be16d4bb052b04/ci-extend+reference.podman.yml ( at date of issue creation)

And called via include+extend

https://git.conesphere.cloud/conesphere/misc/conesphere.gitlab-ci.yml-snippets/-/blob/fca3ee6be5427f022264413916be16d4bb052b04/podman.default.build.yml https://git.conesphere.cloud/conesphere/misc/conesphere.gitlab-ci.yml-snippets/-/blob/fca3ee6be5427f022264413916be16d4bb052b04/podman.default.tasks.yml

Gitlab Env

Expand for output related to GitLab environment info

root@cloudgit:~# podman exec -it gitlab_gitlab /bin/bash root@gitlab:/# gitlab-rake gitlab:env:info System information System: Current User: git Using RVM: no Ruby Version: 3.1.4p223 Gem Version: 3.4.22 Bundler Version:2.5.4 Rake Version: 13.0.6 Redis Version: 7.0.15 Sidekiq Version:7.1.6 Go Version: unknown

GitLab information Version: 16.8.2 Revision: d06a4a0af68 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.13 URL: https://git.conesphere.cloud HTTP Clone URL: https://git.conesphere.cloud/some-group/some-project.git SSH Clone URL: git@git.conesphere.cloud:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers:

GitLab Shell Version: 14.33.0 Repository storages:

• default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell

Gitaly

• default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
• default Version: 16.8.2
• default Git Version: 2.42.0

Results of GitLab application Check

Expand for output related to the GitLab application check

root@gitlab:/# gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.33.0 ? ... OK (14.33.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain not verifying SSL hostname of LDAPS server 'ipa.conesphere.cloud:389' LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 69 users of 100 limit.

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes Tables are truncated? ... skipped All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 35/3 ... yes 40/4 ... yes 380/5 ... yes 45/6 ... yes 45/7 ... yes 35/8 ... yes 47/9 ... yes 45/11 ... yes 45/12 ... yes 45/13 ... yes 35/14 ... yes 35/15 ... yes 380/16 ... yes 35/17 ... yes 45/18 ... yes 40/20 ... yes 35/21 ... yes 35/22 ... yes 47/23 ... yes 53/24 ... yes 51/25 ... yes 51/26 ... yes 53/27 ... yes 53/28 ... yes 53/29 ... yes 53/30 ... yes 53/32 ... yes 53/33 ... yes 53/34 ... yes 53/35 ... yes 53/36 ... yes 53/37 ... yes 53/39 ... yes 53/40 ... yes 45/42 ... yes 45/43 ... yes 45/44 ... yes 45/45 ... yes 53/46 ... yes 57/47 ... yes 51/48 ... yes 53/49 ... yes 35/51 ... yes 57/52 ... yes 57/54 ... yes 57/55 ... yes 57/56 ... yes 57/57 ... yes 35/58 ... yes 47/59 ... yes 45/60 ... yes 57/61 ... yes 53/63 ... yes 53/64 ... yes 123/65 ... yes 53/66 ... yes 92/69 ... yes 35/70 ... yes 57/71 ... yes 53/74 ... yes 322/77 ... yes 57/78 ... yes 57/79 ... yes 47/80 ... yes 53/81 ... yes 90/82 ... yes 122/86 ... yes 53/90 ... yes 35/93 ... yes 35/95 ... yes 92/99 ... yes 124/100 ... yes 127/101 ... yes 53/102 ... yes 92/103 ... yes 45/106 ... yes 45/108 ... yes 45/109 ... yes 45/111 ... yes 92/115 ... yes 45/116 ... yes 127/120 ... yes 120/125 ... yes 245/126 ... yes 245/128 ... yes 245/129 ... yes 245/130 ... yes 245/131 ... yes 245/132 ... yes 245/133 ... yes 245/134 ... yes 92/142 ... yes 53/144 ... yes 245/145 ... yes 122/147 ... yes 45/148 ... yes 45/149 ... yes 57/150 ... yes 122/151 ... yes 57/152 ... yes 245/153 ... yes 45/154 ... yes 131/155 ... yes 90/157 ... yes 53/160 ... yes 92/161 ... yes 92/162 ... yes 92/163 ... yes 45/165 ... yes 45/166 ... yes 45/167 ... yes 53/168 ... yes 45/170 ... yes 45/171 ... yes 57/174 ... yes 57/176 ... yes 92/178 ... yes 92/179 ... yes 92/180 ... yes 45/181 ... yes 302/182 ... yes 92/183 ... yes 92/184 ... yes 92/185 ... yes 53/186 ... yes 92/187 ... yes 45/188 ... yes 53/189 ... yes 53/190 ... yes 53/191 ... yes 245/194 ... yes 53/196 ... yes 45/199 ... yes 53/200 ... yes 45/201 ... yes 45/202 ... yes 45/203 ... yes 53/205 ... yes 47/206 ... yes 45/207 ... yes 53/208 ... yes 92/214 ... yes 53/215 ... yes 45/216 ... yes 92/217 ... yes 359/218 ... yes 92/219 ... yes 92/220 ... yes 45/222 ... yes 365/223 ... yes 45/226 ... yes 53/227 ... yes 53/228 ... yes 45/229 ... yes 282/230 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 3.0.6 ? ... yes (3.1.4) Git user has default SSH configuration? ... yes Active users: ... 34 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

• Maybe switch to API calls if this behaviour works as intended