Enforce CI limits

We start to have more and more CI active users.

This is meta issue to discuss potential limiting of free service with steps to implement that.

Things that I consider that we should do:

Limit total artifacts size per-project:

1. Make this Application Setting configured globally,
2. Make this Project Setting configured by Administrator,
3. Show the total size of artifacts stored in https://gitlab.com/gitlab-org/gitlab-ce/pipelines/settings,
4. Introduce a button to separately delete artifacts for tags and branches,
5. Enforce the limit when trying to push artifacts.

Limit total size of container images:

1. Make this Application Setting configured globally,
2. Make this Project Setting configured by Administrator,
3. Show the total size of container images stored for Project,
4. Enforce the limit when trying to generate token with push credentials.

Limit number of builds executed in parallel for single project:

1. Make this Runner setting to define how many builds can be executed for single project at single time,
2. Extend builds/register.json with query checking number of builds being executed for project,

Everything else

(We could also follow the same for LFS. However, it's much harder to remove data for LFS).

The question is whether this is a proper approach for doing the limits. We could think later about limits that should be applied on GitLab.com.

cc @stanhu @markpundsack @pcarranza

Changed title: CI abusers → Enforce CI limits

@ayufan Small point, but people using up a lot of free resources that are otherwise unrestricted probably shouldn't be called "abusers". :) Abuse is when they use CI for bitcoin mining, or phishing sites, or whatever.

Limiting the number of parallel builds doesn't seem like it would have an impact on our usage, it would just smooth out the usage. The total build minutes would still be the same, right?

I'm not sure how much cost we have from storage (artifact and image), but I'd expect the pipeline runs to be the largest cost, so would love something that would make a difference there. As mentioned elsewhere, disabling redundant builds of gitlab-ce mirrors might be the single largest win. Disallowing runners to pick up mirrors might be the generic solution which allows people to run pipelines on mirrors as long as they provide their own runners. Is that doable quickly? Would it be significant?

@markpundsack

Yes, I know :) I started naming them: active users.

We fixed mirrors. It was very small subset of everything that we run. I expected it to be much more.

"Heavy users"?

Mentioned in issue #22689 (closed)

Mentioned in issue #24344 (closed)

Mentioned in issue #24643 (closed)

Being able to reduce artifact retention via retention policies like #19208 (moved) might also help reduce the storage footprint of Very Active Users.

Also retention policy for container images is a related issue https://gitlab.com/gitlab-org/gitlab-ce/issues/20247.

Milestone changed to %8.15

Added ~127616 ~19492 labels

Adding UX and Frontend because I think this work will need UX and Frontend work. Let me know if that is wrong.

Mentioned in issue #24692 (moved)

Please make this a priority, currently CI artifacts are using ~10Tb of data (beating LFS storage) which is making it quite complicated to maintain and scale.

Added Deliverable label

Just a first thought on this, although it might be only partly useful as gitlab is most of the times on premises. Although it could solve problems for GitLab.com

What about being able to configure a self hosted storage to keep these artifacts and or container images?

This would then be a setting globally or per project or even just in the gitlab.yml file.

it seems my comment is slightly related to https://gitlab.com/gitlab-org/gitlab-ce/issues/20247#note_14900692

I've been assigned to the frontend work here, so please let me know when UX is ready for frontend work to begin @dimitrieh

Reassigned to @dimitrieh

Added ~122770 and removed ~19492 labels

@pcarranza

Will it help if we do S3 uploading for artifacts?

(We can easily enable that with CarrierWave).

@ayufan I think that might help. I think @patricio tested that out as part of https://gitlab.com/gitlab-org/gitlab-ee/issues/846.

Make this Runner setting to define how many builds can be executed for single project at single time,

Should this really be a Runner setting since it is GitLab CE/EE who is scheduling builds?

Reassigned to @ayufan

The notes from our discussion: https://docs.google.com/document/d/1n41Zh4yN5DTKJ_K1RUjyd_NS2JBEGcF1pxvAMn76n1M

We will do this meeting every week.

@ayufan @markpundsack do we still need potential UX on this one? :)

@dimitrieh yes.

@dimitrieh It seems that we will need your help as we will probably work on build minutes.

@ayufan I heard about an upcoming meeting on this subject, should I be part of that?

Added ~19492 label

My issue #18351 (moved) proposes (at least) two concepts relevant to this one:

• Global, admin-controlled artifact expiration time override
• Artifact cleanup button

Google Doc for weekly agenda to discuss: https://docs.google.com/document/d/1n41Zh4yN5DTKJ_K1RUjyd_NS2JBEGcF1pxvAMn76n1M/edit

@ayufan - is there any chance of making the number of builds per project customizable either in the .gitlab-ci.yaml (global/job levels), or at the project setting level (similar to Travis), instead of in the runners config? Having some form of currency handling would be a bonus, but making it user configurable would be even better.

going to remove UX from this as https://gitlab.com/gitlab-org/gitlab-ee/issues/1405 is basically the related issue which needs that

removed ~19492 label

@halberom

Number of builds that runner can process is purely dependent on configuration of your machine or manager so this should always be configuration of runner. I would think about number of concurrent builds that single project can run, but this seems like a different problem to solve :).

@ayufan I second @halberom on its request.

There is two limits here:

• how many build a runner can do in parallel
• how many build a project can do in parallel

1. A runner can be shared in multiple projects. So for example let's say a runner is configured to do up to 50 builds.

2. A particular project that uses external ressources (e.g creating vm on GCE/AWS ) needs to limit it's number of builds. It would be nice to be able to do it and limit like 10 builds. This feature would be similar as mentioned to travis 'builds limit'.

For now the only solution I have to limit the number of build for one particular project, is to create a dedicated runner for it, which add more maintenance / setup.

@ayufan - thanks for responding, I may have misinterpreted the point of this ticket. The section 'Limit number of builds executed in parallel for single project' suggested to me that there was an element of concurrency handling, which would fit within the 'enforce ci limits' plan. If you can define concurrency then by default you are imposing limits which otherwise would be nontrivial as @ant31 pointed out. But maybe this is more of an internal gitlab.com architecture ticket? Thank you for your all your hard work.

changed milestone to %29

This is meta issue to follow the effort that we are doing.

added ~301096 label

removed ~19173 label

so there is right now no simple config to limit running jobs per branch? we have e.g 2 runners that both have the exact same config and tags.

and multiple master commits on a project, which trigger deployment, since there are more runners they pick up thos jobs, which results in multiple parallel deployments being run, which is really bad.

what is the supposed solution? (creating runner for deployments? seems to be a config overhead and also its not fault-tolerant, what if the one runner is out-of-service?)

i would like to do it via .gitlab-ci.yml like:

deployjob:
  exclusive: true

as we could easiliy run unittests in parallel.

mentioned in issue #29065 (moved)

mentioned in issue #23777 (moved)

mentioned in issue #22502 (moved)

Bump / Where did this end up?

It's essential that projects can specify their parallel limits independent from runners.

Will wait for @bikebilly to schedule.

Particularly interested in limiting a specific projects so that it's jobs cannot run in parallel.

added [deprecated] Accepting merge requests label

removed [deprecated] Accepting merge requests label

removed Deliverable label

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue