Estimate the costs of E2E test parallelization
Given that parallelization is likely to increase our usage of CI resources, and therefore increase our costs, it would be prudent to understand our current costs and to estimate how much parallelization will increase those costs.
For a very rough estimate, assuming review-qa-all runs once per pipeline, in 8000 pipelines per month, at a cost of $0.0950 per hour, with a job execution time of 7m 30s (not parallel) or 17m (parallel: 5 jobs), the cost per month would be:
- not parallel: $95
- parallel: $215
That extra cost would save us about 3m 30s in each pipeline (because the instead of waiting 7m 30s for the job to complete, we'd wait only the duration of the slowest parallel job, which is about 4m).
Note: this estimate only considers parallelizing the review-qa-all job, and assumes it will run in every MR. We should also consider the cost of parallelizing package-and-qa
This script could be useful: https://gitlab.com/gitlab-org/quality/toolbox#gather-jobs-durations
Activity
changed milestone to %12.0
added QA Quality backstage [DEPRECATED] labels
There's a handful of estimate of job execution times: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28283#note_171255975
I'll collect some more to give us a ballpark.
-
In that MR 27 tests consistently took around 7m 30s total test execution time and around 17m total job execution time when one test failed, but about 25m when 4 tests failed.
Edited by Mark Lapierre -
From a test branch with parallel and single jobs for comparison: https://gitlab.com/gitlab-org/gitlab-ce/commits/qa-review-knapsack-collect-data
1 job:
Test Job No. of passes No. of failures https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/223796794 5m 51s 7m 41s 28 0 5 jobs:
Test Job No. of passes No. of failures https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/223796789 2m 15s 4m 6s 0 1 https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/223796790 1m 33s 3m 23s 2 https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/223796791 3m 10s 5m 2s 9 https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/223796792 2m 46s 4m 35s 6 https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/223796793 1m 59s 3m 57s 9 total 11m 43s 21m 3s 26 1 There were some retries in the parallel jobs.
Edited by Mark Lapierre -
1 job:
Test Job No. of passes No. of failures https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/224578031 5m 35s 7m 38s 28 5 jobs:
Test Job No. of passes No. of failures https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/224578021 0 2m 5s 0 https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/224578022 43s 2m 33s 2 https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/224578023 2m 9s 4m 8s 9 https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/224578024 1m 51s 3m 45s 6 https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/224578026 1m 11s 2m 59s 9 total 5m 54s 15m 30s 26
mentioned in merge request !28283 (merged)
-
The jobs run on
private-runners-manager-Xrunners, which aren1-standard-2machines (see Slack thread), which cost $0.0950 per hour inus-east1 added Cloud Spend label
-
It's difficult to predict the number of times the E2E test jobs would be run. Even if we assume once per pipeline and count all pipelines, the number of pipelines if fairly consistent on CE, but rose substantially on EE over the last 2 months:
CE: https://gitlab.com/gitlab-org/gitlab-ce/pipelines/charts
EE: https://gitlab.com/gitlab-org/gitlab-ee/pipelines/charts
But for the sake of comparing parallel vs. non-parallel execution, I'll assume the last 2 months were unusual, and so the average number of pipelines on CE and EE is about 8000 (but could be anywhere between 3600 and 17500)
-
For a very rough estimate, assuming
review-qa-allruns once per pipeline, in 8000 pipelines per month, at a cost of $0.0950 per hour, with a job execution time of 7m 30s (not parallel) or 17m (parallel), the cost per month would be:- not parallel: $95
- parallel: $215
That extra cost would save us about 3m 30s in each pipeline (because the instead of waiting 7m 30s for the job to complete, we'd wait only the duration of the slowest parallel job, which is about 4m).
Thanks for the numbers Mark! @meks should we still stick to 5 runners which would be ~ $215 / month?
changed milestone to %12.3
added missed:12.0 label
mentioned in issue #64290 (closed)
changed milestone to %12.1
changed milestone to %12.2
added missed:12.1 label
mentioned in issue #65094 (closed)
mentioned in epic &1276
mentioned in issue #66572 (closed)
mentioned in issue #66594 (closed)
changed milestone to %12.3
added missed:12.2 label
mentioned in issue #66618 (closed)
Hey!
GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current
gitlab-cerepository will become a read-only mirror, without any proprietary code. All development is moved to the currentgitlab-eerepository, which we will rename to justgitlabin the coming weeks. As part of this migration, issues will be moved to the currentgitlab-eeproject.If you have any questions about all of this, please ask them in our dedicated FAQ issue.
https://gitlab.com/gitlab-org/gitlab-ee/issues/13855You can also find more information here:
https://gitlab.com/gitlab-org/gitlab-ee/issues/13304- https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/
Frequently Asked Questions
For an up to date list of questions and answers, please take a look at
https://gitlab.com/gitlab-org/gitlab-ee/issues/13855What will we do with the repository names?
https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.
Why rename gitlab-ce to gitlab-foss?
Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear
I created a merge requests for CE, and this got closed. What do I need to do?
You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.
How does the licensing work in this new setup?
Everything in the
ee/directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in theee/directory, the process of contributing changes is the same as when using the gitlab-ce repository.Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?
No. Merge requests submitted to this project will be closed automatically.
Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?
Yes.
How will this affect users of GitLab CE using Omnibus?
No changes will be necessary, as the packages built remain the same.
How will this affect users of GitLab CE that build from source?
Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.
Where can I see a timeline of the remaining steps?
https://gitlab.com/gitlab-org/gitlab-ee/issues/13304Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?
Yes, these changes will be synced to gitlab-foss automatically, several times per day.
