Originally proposed in https://gitlab.com/gitlab-org/gitlab/-/issues/414574. DRI: @rutshah (formally `@tigerwnz`) Tech Lead: @allison.browne ## Status Update (Apr 28th 2025) We're deprioritizing External CI Jobs initiative through the end of FY26 to focus resources on critical Verify roadmap initiatives instead. Our immediate priority is delivering high-quality experiences in key CI areas. Please leave a comment if this feature/initiative is of interest to you, with the following information * What problems will this feature solve for you? ## Problems to solve We want to provide a better way for the GitLab CI pipeline to integrate with external CI/CD tools. ### Problem#1: Integrating a GitOps Deployment Tool (Flux) with the Gitlab CI pipeline (Validated by @nagyv-gitlab ) It is difficult to combine pull-based deployments with traditional CI/CD pipelines, due to differences around what is considered "done". For example, a (simplified) traditional deployment might happen like this: 1. Code is pushed to `main` 2. Pipeline "pushes" new code to infrastructure Here, the state of the infrastructure depends on the state of the pipeline - if the pipeline is successful, the new code is deployed. In a GitOps deployment, the direction is reversed: 1. Code is pushed to `main` 2. Infrastructure "pulls" new code and applies changes In this scenario, the infrastructure state depends on the state of the repository, which creates a problem when combined with a pipeline: The pipeline runs when code is pushed to the repository, but at this point we don't know the current reconciliation state. A pipeline must then poll the infrastructure to check if the desired state has been reached before it can proceed with additional tasks. See https://gitlab.com/gitlab-org/gitlab/-/issues/405007 for additional details. ### Problem#2: Improved integration between Gitlab CI Pipeline and external systems that currently use Commit Status API The primary integration use case we want to solve is with Jenkins **Jenkins**: Customers have various hurdles in migrating Jenkins to Gitlab CI. One challenge could be taking an 'all-or-nothing' approach while migrating from Jenkins to GitLab. This epic provides an option to migrate to Gitlab in partial steps where hard-to-migrate parts of the pipeline can continue to run in Jenkins. With this, we can unlock the ability for users to use other GitLab features (Security, Compliance) sooner regardless of where they are in the migration journey. ## User experience goal A CI pipeline can contain jobs that represent a process that is executed by an external process, instead of a GitLab Runner. a) This will enable "hybrid" pipelines, that combine traditional CI tasks (for example, running tests) with Kubernetes deployment operations (waiting for the desired state before proceeding). b) Call Jenkins Jobs from Gitlab pipeline, get the response back and that could influence whether that job passed or failed. 1. pipeline is created and starts running 2. when the external job is ready to be executed we put it in a pending state. 1. Optionally we could fire a webhook to the external system (if configured) to notify if a job is waiting and can initiate the processing. 2. the external system notifies the pipeline's job that the processing has been completed with status success or failure. The job will fail if there is no response within the specified timeout period in the external job configuration 3. the pipeline resumes or stops. ## Technical summary Jobs defined in a `.gitlab-ci.yml` file can be described as `external`, which represents a process that is not executed by GitLab. These jobs are configured and behave in much the same way as traditional jobs, however instead of being picked up by a runner, an API is provided that allows an external service to set the outcome of the job, at which point the pipeline continues as normal. ## Outcomes - Syntax exists in a GitLab CI/CD template to allow setting individual jobs as external. There is no further configuration, all jobs are either not external (as they are today), or external. External jobs cannot have a `script` defined. - An external job is not picked up by a runner. When started, it enters a new state and remains there until either the timeout is reached, or a valid job result is provided via the API (see next point). - An API exists that allows setting the status of an external job to either "success" or "failure". When an external job receives a status from the API, the pipeline continues (or doesn't) in the same way as non-external jobs that return the status. - Jobs with `external:true` are jobs that become `waiting_for_callback` instead of `pending` when they are unblocked, at which point an appropriately formed call to the commit status API is required to move the job forward to a different status. - Based on https://gitlab.com/groups/gitlab-org/-/epics/10866#note_1880965329 ## Implementation 1. [Create job/pipeline status to represent waiting for external input](https://gitlab.com/gitlab-org/gitlab/-/issues/415736) 2. [API endpoint(s) for updating external jobs](https://gitlab.com/gitlab-org/gitlab/-/issues/415738) 3. [Add external job CI syntax](https://gitlab.com/gitlab-org/gitlab/-/issues/415737) 4. Feature flag cleanup