As a user I would like to create a component that uses a container image that is also built during the release process of the component itself. For example:
I tag version 1.0 of the repo.
A tag pipeline runs and build the Docker image my-image:1.0
The component yaml file uses by default the image that matches the same tag.
This problem cannot be solved using predefined CI variables because those are evaluated in the context of the consumer project, not the current file being evaluated.
Proposal
We can make the component ref related to the current file context available for interpolation. For example:
The current workaround would be to either specify the component version twice: once in the include of the component and another as input for the component that is then being used to reference the image.
Another option is to reference the image version in the pipeline template file in a commit just before creating the tag. So that the git revision content hardcodes the tag.
Technical Details
Currently, we skip the interpolation process if an included CI config file has no header. We do this to save on processing time because interpolation is expensive, and so far it's been okay because the only values available for interpolation are inputs and they must have a header. However, the new context interpolation option does not require a header so we need to remove this check. Performing interpolation on all included CI config files may significantly degrade performance, so we need to instrument interpolation to ensure this doesn't happen. If we can remove the CI header check without degrading performance, we can implement $[[ context.ref ]] as described in this issue. Otherwise we'll need to find another solution.
@timofurrer I created this issue from your use case. Would this solve the problem?
@dhershkovitch I believe we had other customers asking for a similar functionality where we want to reference information relative to the "current included file" rather than the pipeline project. What do you think?
@fabiopitino it would also be great to have the $CI_REGISTRY_NAME (and similar) available in such a context - so that it's easier to reference the entire image in the component.
@fabiopitino this is one reason we would like to release steps primarily as OCI images. We can include binaries / the container in the payload. And the step definition can be in the metadata. (@marshall007's idea)
Parse the version from - component: mygitlab.com/group/project@0.0.1Use the value version 0.0.1 as an input for image: "registry.mygitlab.com/group/project:???"
I'm curious, could this also be expanded to allow users to reference files inside of the component's project?
A use case would be a component that need some configuration files and those files are the same where the project is included. For example, I set up a component in our instance used to quickly add clang tools analysis (clang-tidy and clang-format) to all our C++ projects and since all those projects share the same settings (for clang), it would have been nice to reference those directly from the component.
The workaround I found, for now, is to have a different project (it could be the same project used for the CI component to be honest), and each job downloads the needed files via cURL.
I have use cases where I want to fetch some scripts from my component, so either by the API or by a git clone. But I need first to know these contextual information (that the runner knows).
Here what I want to do in my component:
my-component-job:script:-echo "My Component version $[[context.version]], sha $[[context.sha]]"-git -C .component-name clone --no-checkout --filter=blob:none -b "$[[context.branch]]" https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.org/$[[context.project.full_path]] .-git -C .component-name sparse-checkout init --cone-git -C .component-name sparse-checkout set --cone scripts-git -C .component-name checkout $[[context.branch]]# question: how to sparse-checkout precisely on $[[context.sha]]?-.component-name/scripts/my-bash-script.sh-python .component-name/scripts/my-python-script.py-pipx run .component-name/scripts/my-pep723-script.py
For fetching a simple file we can use the Gitlab File API. It also works to retrieve a folder using the archive API, but in every case, we need to kown the correct ref:
my-component-job:script:-echo "Fetching configuration file for Component version $[[context.version]], sha$[[context.sha]]"- PROJECT_PATH="$[[context.project.full_path]]"- REF=$[[context.sha]]- curl --header "PRIVATE-TOKEN:$CI_JOB_TOKEN" "https://gitlab.org/api/v4/projects/${PROJECT_PATH//\//%2F}/repository/files/config.file?ref=REF"
So I would needed these information in the context:
context.project.full_path: to perform the git clone
context.branch: git clone need a tag or a branch to clone
context.sha: ideally I want to sparse-checkout only on this precise sha1
context.version: I would like to be able to display the full version in the log
This is also the case with "config" files, to fetch (using curl) at the exact same version that the CI component, we need to know the sha of the commit.
Without this information fed by the runner, there is no way to act in a MR or on master (without tag), simply because a component cannot know its current sha1. It can hardcode its current branch (commits before HEAD are not reproductible) or hardcode the future tag ("commit tag" solution), but this is not "seamless" for debugging branches.
@dhershkovitch@avielle Note that some use cases such as this one could be solved by CI steps. Any problem related to "adjacent files" will almost 100% be resolved by CI steps. With CI steps the Runner will checkout the whole step directory.
Looking at some other interpolation issues and level of complexity I would say this issue would be weight 4. We likely need a feature flag for the rollout as well.
@DarrenEastman@josephburnett@ayufan I want you to be aware of this issue, which is something that our users will probably expect to be supported with steps as well
@dhershkovitch this issue seems to be about having execution context available to parameterize the component and it's inputs. Steps already has this in the ${{ }} form (expressions) which is evaluated at the last possible moment so it can take into account outputs and environment variables from previous steps (or files or whatever).
Problem they are trying to solve: We have a component that has other files in its project that we'd like the component to get at runtime, but we don't know the commit id or version of the component at runtime so we can't get the right files. Context variables, such as $[[ context.project.full_path ]] and $[[ context.ref ]] which are suggested in the issue would solve our problem.
@dhershkovitch@fabiopitino hey y'all, I've started working on this and I'd like to make sure we're aligned on the feature proposal. Is this the complete list of values we'd like to start with?
project.id
project.full_path
sha - do we have sha and ref for all include types? I'll have to double check
ref
version - we'll only have this for components, not other include types. What is the use case for this one? The only one I can think of is including components of the same version from the same project, and I'm not sure that happens often in the wild
@avielle@dhershkovitch The primary goal of this issue (the problem statement) is to have the ability to reference container images from the component project rather than hard-coding values.
This problem might be resolved in its entirety by CI steps and by how we decide to build and publish CI steps that depend on a container image. @josephburnett to confirm.
If after CI steps we still have edge cases on this topic we can explore this issue more.
@fabiopitino thank you for providing this context. I'm going to unassign myself from this issue while we figure out if the problem described in it can be resolved using Steps
Sad to here this has been descoped. When Gitlab Step is supposed to land? Basically now we (as pipeline/components developers) are stuck where CI Component is cool but lack the access to the SHA-1 of the very component, to retreive ourself additional files at the exact point when the component has been included. So CI Component is not really different from classic "include".
Today, I can only do the checkout on the HEAD of the current branch (hardcode ie master), and try to test on alternative branches for pre-engineering checks ($CI_COMMIT_SHA).
Basically, if you can only provide context.sha1, that would allow us to wait for the real solution "Gitlab Steps" in 2025 or 2026.
As you've highlighted in #438275 (comment 1911690118) - Steps don't solve the issue of the versioning - I assume that a single step will never have the context of the "outer" layer - that is the component itself. So I don't think it would help much.
@fabiopitino@avielle what if we scope this issue to only items that we probably wont be able to achieve using CI step, I believe we wont be able to extrapolate version number, which is the initial ask for this capability, users would like to:
know which version is running in case they use shorthand or ~latest
Match the same image version to the component number (I would assume this is applicable for CI steps as well)
so I propose we create an issue to interpolate version number only for now
@dhershkovitch this would be awesome and the "only" urgent thing I need for the components I maintain (mostly OpenTofu).
Regarding:
know which version is running in case they use shorthand or ~latest
I think that ~latest should resolve to an actual tag name - otherwise I'd be quite useless. Or at least resolving it to latest (without the tilde) - so that it's easy to pull in other assets using latest as a version (like a container image). If one needs the tilde they can add it back, but it would with the current capabilities be almost impossible to remove it.
An increase in interpolation time will affect the loading and processing of external configs, so we can use the json.config_external_process_duration_s metric to monitor it. I've added a graph to the Pipeline Creation dashboard in Elastic to make the monitoring easier. I can't link to it directly but it looks like this:
The current daily median has varied between 0.45s and 0.55s over the past week
@marknuzzo@dhershkovitch the next step will be to remove the conditional header check under a feature flag and see if those bars get bigger Stay tuned
Call with customer about interlinked components and inline structured. Ex: AWS component and multiple templates - configure template needs to refer to the version of the deploy version. Today we need to check in and it defeats the purpose.
Component telemetry is super important - components are being built on top of each other - and hard coded versions make that hard to maintain.
NOTE: If the monitoring shows that we can't implement this feature without a significant performance hit, we'll need to discuss how to proceed. We might need a spike to figure out another way to implement it
This is similar to an issue I submitted, #502247. (I didn't find this issue in my search when I created mine.) From reading through this issue, it doesn't seem like its resolution will completely address my use case though. Can someone help me understand where this issue currently stands and what feature(s) will be available once it's resolved?
I am another Gitlab Ultimate customer interested in this feature. Specifically to reference container image version which is built within the component and version needs to be referenced by template
@nmezzopera I'm unassigning myself from this issue because it's paused
What I got done
I added a metric that allows us to monitor how long it takes to process a config loaded with include:component. This issue could severely impact that performance, so we'll need that metric to do our due diligence before we proceed with the implementation
What needs to be done next
The next step is to remove the existing check that aborts interpolation unless the included component file has a CI header. Removing that check is a requirement for this issue, and it's what could lead to increased performance demands. The check should be removed, the performance monitored for a week, and then we can proceed with the implementation
Customer want the image tag and component to be in the same version.
Ideally they have a repo with component (gitlab yaml files) and source code for the image.
They create a tag 1.2.3 and that spawns a tag pipeline:
They can then build the image and push to registry using CI_COMMIT_TAG
The yaml files use "$[[ context.ref ]]" (effectively the same value as CI_COMMIT_TAG)
When component 1.2.3 is included then image inside the component is resolved to 1.2.3
Today, inside the component you have something like:
image: whatever:1.2.2
They want to use context.ref which is resolved by gitlab to component version
They can't do the above because tags are immutable, so they can't create a tag 1.2.3 and spawn a pipeline that updates code for component and publish image
Business Impact
If this capability is not available, it will block their CD adoption plans for fully automated deployments to production environments next year within the customers org. Customer is currently about to consolidate more than 10 different CD solution into GitLab.
Request
Could we please review the current priority status of the implementation? We would like to get back to the customer with a high level estimate for availability / our plans.
If we want to talk with the customer on details, the customer DRI is more than happy to get on a call with us for more discussion / insights. Just let me know here or in SLACK.
Having feedback on this would be very helpful for our customer. Thank you!
Today, we have spec:inputs:xyz and we can interpolate $[[ inputs.xyz ]]. Now, we want to interpolate $[[ context.abc ]] and it feels like we need spec:context. Or even this;
spec:context:[ref,version]
Yet, I am also in favor of continuing this conversation in #438275 and let this MR merged to test the performance.
@nmezzopera@jreporter - @furkanayhan and I have discussed some options for solving this issue and think that we should keep the header presence check and solve it with another keyword, like suggested above:
spec:context:somethinginputs:env:...---
If you're fine with this, we can schedule the issue on 17.10 - it might be worth it to spike first.
Let's also give some context to explain why we made this decision.
If we started interpolating every YAML file and tried to find the special brackets ($[[ something ]]), this would be an implicit opt-in behavior. Do we want this? (maybe a risk of a breaking change)
As we don't allow any other inputs that is not defined in spec:inputs, we should also not allow any "context" that is not defined in spec:context.
If we started interpolating every YAML file and tried to find the special brackets ($[[ something ]]), this would be an implicit opt-in behavior. Do we want this? (maybe a risk of a breaking change)
@furkanayhan This decision was already made when the spike was first talked about - and it was going to be an implicit opt-in behavior. You can read more info in the threads provided in the MR.
After much thought, we can simply iterate with both solutions since they can be done in parallel and the MR for the presence check is pretty much complete. I will focus on getting that out the door and monitoring the performance, since it doesn't hurt to have data and see if this solution will even work the way we want to (which is why we wanted to do this in a ff).
We can then write up another issue to spike on the solution you suggested, so that @nagyv-gitlab can start getting some context, since it's more efficient to spike on this rather than wait due to our current team situation, I'd say.
@nmezzopera - we might need to squeeze in a spike issue with Furkan's suggestion that also blocks this issue. wdyt?
Please feel free to redirect my questions to another issue if this is not relevant... @lauraXD@furkanayhan
Do we know how someone is going to know when to use spec:context and when it is not appropriate to that? I am mostly concerned with performance and if there are certain guardrails that need to be added to ensure that this is being used as intended
If we started interpolating every YAML file and tried to find the special brackets ($[[ something ]]), this would be an implicit opt-in behavior. Do we want this? (maybe a risk of a breaking change)
Would this be managed by a setting? How should I go about validating this use case with customers? What are the benefits of brackets? What is the risk to how long a pipeline would be running as a result of this behavior?
This problem cannot be solved using predefined CI variables because those are evaluated in the context of the consumer project, not the current file being evaluated.
@lauraXD@furkanayhan Thank you for all the discussion here. Let's see the metrics coming in and stop here for now.
We have a sensible workaround of providing the image tag as an input. It's already used widely in the published components, and it likely does not block adoption severely. Let's focus our attention to other areas for the coming months, like self-managed component support.
Hello. I still need to pass the ref of the tag as input and it is ugly, error prone, and and does not allow to set sha1 as ref. This is not practical, especially when testing change (MR) on the component in another project.
thanks @furkanayhan - are you able to add color to my other question:
Do we know how someone is going to know when to use spec:context and when it is not appropriate to that? I am mostly concerned with performance and if there are certain guardrails that need to be added to ensure that this is being used as intended
Do we know how someone is going to know when to use spec:context and when it is not appropriate to that? I am mostly concerned with performance and if there are certain guardrails that need to be added to ensure that this is being used as intended
@jreporter It's too early to answer this question because we don't have a proposal to work on at the moment. However, we should keep this concern in mind when working on this issue. Thanks!