CI jobs with callbacks
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Proposal
I have encountered a situation where I need a CI job to trigger a specific external process, but there is no point in having the job wait around after the external process is triggered. However, I need to delay subsequent jobs in the pipeline until that external process is complete. The external process has the capability of posting back to a specific URL when its work is complete. My current approach to solving this problem is to split the job into two, with the second job being a manual job that will be triggered by the external process utilizing a PAT/GAT; that releases the runner, but it requires a long-lived credential to make it work.
What I propose instead is to create a new YAML keyword, say, async. If async is true, then gitlab would inject a new, masked CI variable, CI_JOB_CONTINUE_URL. Once the job script completes, the runner is released, but the job goes into a "wait" state. During this "wait" state, any HTTP request to the CI_JOB_CONTINUE_URL would cause the job to complete successfully--or, if the HTTP request is a POST containing a JSON payload, the result of the job can be set according to the contents of that payload (details to be determined). If the job timeout is reached, then the job would go to a failure state. Of course, the CI_JOB_CONTINUE_URL value would contain the appropriate details, such as the project ID, pipeline ID, and job ID, as well as a one-time-use token to authenticate. If the URL is hit a subsequent time, or hit with an unknown token, then nothing happens.
This would solve my use case: I only need to create one job. It would also prevent accidental running of the manual job before the external process completes. It would also provide an appealing alternative to a job that polls the external process where that external process has a means of calling a callback when it is done. Finally, it improves the security posture, as this scheme avoids needing an API token to continue the pipeline.