[POC Discussion] Parent-Child Artifacts CI Functions POC
Description
This issue covers the progress and requirements of the highly-requested Parent-Child Job Artifact reference feature.
POC Walkthrough
POC Walkthrough (Passcode: 1G4qdz!7) - GitLab Internal only for now.
Feature Support
| Feature/Requirement | Support Status | Explanation |
|---|---|---|
| Parent - Child Pipelines | Fully supported | |
| Dynamic Pipelines | Fully supported | |
| Nested Pipelines (Multi-level parent - child) | Need to configure the in-between pipeline to fetch the artifacts from the lowest level and expose the artifacts so that the top level can pick it up | |
| Cross-Project Pipelines | CI Job token granular permissions may not support cross-project configuration |
Implementation Details
POC Project: https://gitlab.com/panoskanell/test-parent-child-artifacts-private.git
Since the script was too large, I had to split it in 2 functions (Ideally should be even more) and wrapped them in a CI Component.
CI Component: download-child-artifacts
| Type | Name | Description |
|---|---|---|
| Argument | bridge_job_name |
Specifies the pipeline from which to fetch artifacts |
| Argument | job_names |
Clarifies which jobs to fetch artifacts for |
| Function | find_bridge_job_pipeline_id |
Identifies the downstream pipeline ID for the given bridge job |
| Function | download_artifacts |
For the returned pipeline_id, downloads artifacts for specified jobs (or all jobs if none specified) |
What's missing
- Add ci_job_token support and fine grained permissions for
:id/pipelines/:pipeline_id/jobsand:id/pipelines/:pipeline_id/bridges - Proper testing setup/configuration
Developer's thoughts
- Usage looks quite simple to integrate so it should provide a nice user experience with clean YAML files
- This solution works fine and is very open ended from its' nature. Customers may want to modify the implementation of this feature in order to add logs for debugging or modify the current functionality which will be possible if we provide the component on a public repo.
My take would be to host this base-level component template on a component repo and make it available for users to use as-is or let them fork it if someone wants to extend/modify it.
This will lead to great flexibility, less feature requests but we will still have to maintain the base template to a certain extent. We would need to decide on conventions as to what we may support/maintain from our end.
AI Native
This POC also showcases Gitlab's potential to become an AI Native Development product, imagine having flexible APIs that allow users to build their own functionality. The same can apply to Gitlab's own AI, it will be able to build steps and functionality for customers/users.