At the moment, on Gitlab.com there is no option to increment the max artifact size for child pipelines. My company workflow uses child/dynamic pipelines to generate dynamic jobs based on certain factors, and this process occurs in a job where we also install some dependencies, enlarging the artifact size and thus not permitting us to consume the newly generated yaml file to trigger a child pipeline from.
The proposal is to add an option to change this limit, or to make this limit higher.
This change would allow there to be a more lax limit, and thus more flexibility when writing our pipelines & jobs.
@joshlambert - in relation to our consumption effort is there any reason why we wouldn't implement this from a business POV (I would expect to add this limit support only in self-managed)
@jreporter - I would imagine we would still want some limit on GitLab.com, otherwise it may be a vector for abuse or some other kind of attack.
The other aspect to think about would be how easy it is to manage these resulting pipeline artifacts. Are they still subject to expiration policies? If you have an overly generous policy, and want to clean some up, how easy is that?
@joshlambert the proposal is to let people set a limit higher than today. I would not want us to set an exorbitantly high limit for artifacts on GitLab.com
We already have a workstream in Category:Build Artifacts dedicated to making artifacts easier to manage. I just wanted to confirm there weren't other changes related your project that implementing this higher limit for self-manage would conflict.
@jreporter@jocelynjane the limit of 1 GB on Gitlab.com is for artifacts upload. The hard-coded limit of 5 MB is for dynamically generate CI configs that are being read in memory on the server side. We set that limit as MVC which allows us to be comfortable with reading 5 MB of files in order to merge CI configurations in memory: !88104 (comment 956698170)
We could definitely allow admins to change this limit as long as we also make them aware of the risks.
That makes a lot of sense thanks @fabiopitino for confirming. @jocelynjane - definitely worth prioritizing in the near future. I did have a recent call that discussed these limits as being non-starters.
There is a GitLab Premiumself-managedcustomer who is also interested in having that limit be configurable as the current behavior requires them to manually adjust that limit in the source for their CI/CD pipelines to work properly. GitLab team members with access to Zendesk can learn more in the ticket.
@morefice We should introduce this at the instance level so that self-managed admins can choose what their limit is.
This issue was initially for SaaS which means the customer would like us to increase the limit of our SaaS instance, but we'll need to ensure that it is safe before we do so. I think a good iteration is to make this more easily configurable by admins instead of hardcoded, and then we can determine if we should increase that limit for SaaS.
Agree with @shampton. Is the proposal to keep the existing size as default and allow for the changes? @gdoyle adding you for awareness and potential Design input (longer term, does it make sense to have this as part of an admin panel?).
We should introduce this at the instance level so that self-managed admins can choose what their limit is.
Agreed to introduce this feature at the instance level as well.
Is the proposal to keep the existing size as default and allow for the changes?
I would suggest to set the default size with the existing limit we have today.
adding you for awareness and potential Design input (longer term, does it make sense to have this as part of an admin panel?).
If we add this setting at the instance level, I think we don't need a design for this first iteration as users administrator will customize the limit as follow:
@morefice Just so I understand, will this only be available in the API at first for instances? Also, can we bake in some instrumentation for this since it's a new feature? Specifically the count of users who customize the setting so we can ensure that it is being used. Depending on usage metrics, we could always spend time adding this into the UI and iterating on it to support more! @jocelynjane@shampton
@morefice thanks for the clarification! I'm okay moving forward and seeing how the instrumentation plays out so we could then iterate on the feature further.