At the moment, on Gitlab.com there is no option to increment the max artifact size for child pipelines. My company workflow uses child/dynamic pipelines to generate dynamic jobs based on certain factors, and this process occurs in a job where we also install some dependencies, enlarging the artifact size and thus not permitting us to consume the newly generated yaml file to trigger a child pipeline from.
The proposal is to add an option to change this limit, or to make this limit higher.
This change would allow there to be a more lax limit, and thus more flexibility when writing our pipelines & jobs.
@joshlambert - in relation to our consumption effort is there any reason why we wouldn't implement this from a business POV (I would expect to add this limit support only in self-managed)
@jreporter - I would imagine we would still want some limit on GitLab.com, otherwise it may be a vector for abuse or some other kind of attack.
The other aspect to think about would be how easy it is to manage these resulting pipeline artifacts. Are they still subject to expiration policies? If you have an overly generous policy, and want to clean some up, how easy is that?
@joshlambert the proposal is to let people set a limit higher than today. I would not want us to set an exorbitantly high limit for artifacts on GitLab.com
We already have a workstream in Category:Build Artifacts dedicated to making artifacts easier to manage. I just wanted to confirm there weren't other changes related your project that implementing this higher limit for self-manage would conflict.
@jreporter@jocelynjane the limit of 1 GB on Gitlab.com is for artifacts upload. The hard-coded limit of 5 MB is for dynamically generate CI configs that are being read in memory on the server side. We set that limit as MVC which allows us to be comfortable with reading 5 MB of files in order to merge CI configurations in memory: !88104 (comment 956698170)
We could definitely allow admins to change this limit as long as we also make them aware of the risks.
That makes a lot of sense thanks @fabiopitino for confirming. @jocelynjane - definitely worth prioritizing in the near future. I did have a recent call that discussed these limits as being non-starters.
There is a GitLab Premiumself-managedcustomer who is also interested in having that limit be configurable as the current behavior requires them to manually adjust that limit in the source for their CI/CD pipelines to work properly. GitLab team members with access to Zendesk can learn more in the ticket.
@morefice We should introduce this at the instance level so that self-managed admins can choose what their limit is.
This issue was initially for SaaS which means the customer would like us to increase the limit of our SaaS instance, but we'll need to ensure that it is safe before we do so. I think a good iteration is to make this more easily configurable by admins instead of hardcoded, and then we can determine if we should increase that limit for SaaS.
Agree with @shampton. Is the proposal to keep the existing size as default and allow for the changes? @gdoyle adding you for awareness and potential Design input (longer term, does it make sense to have this as part of an admin panel?).
We should introduce this at the instance level so that self-managed admins can choose what their limit is.
Agreed to introduce this feature at the instance level as well.
Is the proposal to keep the existing size as default and allow for the changes?
I would suggest to set the default size with the existing limit we have today.
adding you for awareness and potential Design input (longer term, does it make sense to have this as part of an admin panel?).
If we add this setting at the instance level, I think we don't need a design for this first iteration as users administrator will customize the limit as follow:
@morefice Just so I understand, will this only be available in the API at first for instances? Also, can we bake in some instrumentation for this since it's a new feature? Specifically the count of users who customize the setting so we can ensure that it is being used. Depending on usage metrics, we could always spend time adding this into the UI and iterating on it to support more! @jocelynjane@shampton
@morefice thanks for the clarification! I'm okay moving forward and seeing how the instrumentation plays out so we could then iterate on the feature further.
Just a thought - if the 5MB limit is in place to protect memory usage on gitlab.com, due to the need to read the artifacts package into memory before extracting and merging the YAML, would an alternative solution to increasing the limit be to allow the YAML to be made available to the downstream pipeline via a separate mechanism to that used for the rest of the artifacts?
A premium SaaS customer (ZD internal link) is blocked from building an AWS EKS stack to deploy with the CDK due to the requirement to bundle dependencies along with the dynamic CI/CD YAML, exceeding the 5MB limit.
Saw some labels were removed. Is this feature still viable? Why wouldn't this be available to all tiers (specifically with GitLab-CE self-hosted, where computing resources are managed by the customer)? This feature is specifically critical for large monorepos where multiple projects/packages are built from one repository and the jobs/pipelines that are generated are dynamic, because not every project/package needs to be build with every push if there were only changes on some projects/packages.
This would be very useful for us too (Premium user). We are using a pipeline to clone a another repo (Job 1), generating a child pipeline yml from that repo (Job 2) then executing a trigger job based on that yml (Job 3). I tried to combine Job 1 and Job 2, but since the trigger job needed the artifact archive containing the yml file to be under 5 MB, I cannot do this. However, I am curious if a good workaround for GitLab would be to simply download the desired yml file from the artifact archive directly into the trigger job directly instead of needing to download the entire archive (which is what seems to be causing the need for a 5 MB max archive size)?