Make cross-repo CI triggers first-class

Description

Before implementing a full multi-project pipeline, let's start by utilizing our existing trigger functionality, and make it a first-class citizen in .gitlab-ci.yml.

As a developer, I want to be able to trigger CI build and test on one component after completing tests on another component. e.g. make a change to the API, then trigger tests on the front-end to make sure the front-end integration tests still succeed.

Initial Proposal

We currently support triggers via the API, but to use it in a .gitlab-ci.yml requires ugly CURLing and passing secure tokens. We can simplify that process by providing some higher-level construct. Total rough draft:

test_front_end:
  stage: test
  trigger:
    project: gitlab-org/front-end
    vars: 
      api_ref: $CI_BUILD_REF
  only:
  - tags

I'm not sure how passing references should work. It's probably common to need to know the ref of the upstream build, so maybe we just pass that automatically with some convention. But we probably need to support passing other variables too.

One problem with this is the token. The current API expects you to have pre-generated tokens for the downstream project, but as a user, I'd like to be able to just specify the project name and let some other permission system allow the transaction.

Links / references

• Originally from gitlab-ce#15655 (Multi-project pipeline)
• Some discussion in gitlab-ce#3743 (GitLab Pipeline)
• Related feature proposal: gitlab-ce#17069 (Dependencies)

Added ~19173 label

Added alternate proposal from https://gitlab.com/gitlab-org/gitlab-ce/issues/3743#note_5111288.

for alternate proposal, and also for supporting both.

@markpundsack please focus this issue on making the existing (downstream) trigger first class. Upstream dependencies are a separate issue. Please update the description and spin off a new issue for the dependencies as asked in point 5 of https://gitlab.com/gitlab-org/gitlab-ce/issues/3743#note_5114395

Added ~14213 label

Good point @sytses!

Removed alternative proposal and created gitlab-ce#17069 to track dependencies proposal.

@markpundsack

It's ok. It can be handled internally without a need to create or use a token. The only concern that I have is that we need to define how we enforce permission to perform that trigger. There are a few cases:

1. User performs a push/a tag, he creates a set of builds. We can check if that user has a permission to trigger that project. I think that it's fine to say that we should do it only for people who have a push access to specific branch/tag.
2. We have trigger API which is user-less. This is a little bigger problem, because we don't know of who to check the permissions. Maybe if we could somehow define dependent projects that? It could also solve this problem: https://gitlab.com/gitlab-org/gitlab-ci/issues/224

Thanks @markpundsack

Milestone changed to %29

We had opportunity to talk with @ayufan about implementation details, and we should be able to handle it internally if needed,

Added ~122770 label

Any milestones for this? We're heavily needing it :(

added ~724481 label

added ~698938 label

added gitlab-ce~1672341 label

@markpundsack Why it's a only ee feature? I can understand for https://gitlab.com/gitlab-org/gitlab-ee/issues/1681 which is an improvement designed for big projects but this is just a syntactic sugar, not really a feature.

@markpundsack Are you able to respond regarding the ~"EE Premium" label? https://gitlab.com/gitlab-org/gitlab-ce/issues/16556#note_27537724

@alexispires @MrChrisW As an advanced CI feature, at least part of multi-project pipelines will be EE Premium. For example visualizing the links between pipelines will be EEP. It's possible that some of the underlying functionality will be available at lower tiers. Implementation-wise, it may not be worth making .gitlab-ci.yml syntax different between CE and EE. However, we do need to evaluate the use cases here, amongst many factors, so for now, it's labeled with ~"EE Premium".

changed milestone to gitlab-ce%"9.3"

added gitlab-ce~992791 label

assigned to @ayufan

added gitlab-ce992792 and removed gitlab-ce992791 labels

Unfortunately, this seems unlikely to be done for gitlab-ce%"9.3". Changed the label to gitlab-ce~992792.

changed milestone to gitlab-ce%"9.4"

removed gitlab-ce~992792 label

removed assignee

changed milestone to %29

https://gitlab.zendesk.com/agent/tickets/79333

added ~1672340 label

marked this issue as related to gitlab-ce#39640

added gitlab-ce3857543 gitlab-ce3713900 and removed ~1672340 labels

Instead of this push model, I think the pull/subscribe model suggested in https://gitlab.com/gitlab-org/gitlab-ee/issues/6473 would be preferable.

added gitlab-ce~3103453 label

added to epic &414 (closed)

marked this issue as a duplicate of #1681 (moved)

closed

I've merged this issue into discussion of #1861 (closed)

mentioned in issue #1681 (moved)

removed from epic &414 (closed)

@jlenny How is that related to https://gitlab.com/gitlab-org/gitlab-ee/issues/1861?

Typo - it was intended to me marked as a duplicate of https://gitlab.com/gitlab-org/gitlab-ce/issues/52187.

added 1 deleted label and removed 1 deleted label

added Enterprise Edition label