GitLab CI/CD scalability improvements blueprint
What does this MR do?
This merge request adds to docs an architectural blueprint about improving GitLab CI/CD scalability and CI/CD data storage model.
See https://gitlab.com/gitlab-org/architecture/tasks/-/issues/5
Merge request reports
Activity
added devopsverify grouppipeline execution labels
added Architecture Evolution Blueprint label
added documentation label
2 Warnings This merge request does not have any assignee yet. Setting an assignee clarifies who needs to take action on the merge request at any given time. This merge request does not refer to an existing milestone. 2 Messages We are in the process of rolling out a new workflow for adding changelog entries. This new workflow uses Git commit subjects and Git trailers to generate changelogs. This new approach will soon replace the current YAML based approach. To ease the transition process, we recommend you start using both the old and new approach in parallel. This is not required at this time, but will make it easier to transition to the new approach in the future. To do so, pick the commit that should go in the changelog and add a
Changelog
trailer to it. For example:This is my commit's subject line This is the optional commit body. Changelog: added
The value of the
Changelog
trailer should be one of the following: added, fixed, changed, deprecated, removed, security, performance, other.For more information, take a look at the following resources:
https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1564
- https://docs.gitlab.com/ee/api/repositories.html#generate-changelog-data
If you'd like to see the new approach in action, take a look at the commits in the Omnibus repository.
This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge. Documentation review
The following files require a review from a technical writer:
doc/architecture/blueprints/ci_scale/ci_builds_cumulative_forecast.png
doc/architecture/blueprints/ci_scale/ci_builds_daily_forecast.png
doc/architecture/blueprints/ci_scale/index.md
The review does not need to block merging this merge request. See the:
-
Metadata for the
*.md
files that you've changed. The first few lines of each*.md
file identify the stage and group most closely associated with your docs change. - The Technical Writer assigned for that stage and group.
- Documentation workflows for information on when to assign a merge request for review.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
DangerEdited by 🤖 GitLab Bot 🤖- Resolved by Grzegorz Bizon
Previous version of the blueprint is available here: gitlab-com/www-gitlab-com!54748 (closed). We decided to change our strategy a bit, and start with partitioning instead of data retention policy.
- Resolved by Grzegorz Bizon
- Resolved by Grzegorz Bizon
- Resolved by Grzegorz Bizon
- Resolved by Grzegorz Bizon
- Resolved by Andreas Brandl
- Resolved by Grzegorz Bizon
added sectionops label
- Resolved by Grzegorz Bizon
Typically, the main and first question we'd need to answer with partitioning is - what partitioning key do we choose and what is the trade-off (how does it help us scale, what are the drawbacks)?
Previously, we've dug into the application behavior and identified 'access patterns' - i.e. what keys do we typically use to access data. An ideal partitioning key is always present when we query data. If that is not the case, we're likely to end up with a trade off having a "good" way of accessing data (using the partitioning key) and a "much worse" way (crossing partitions).
I understand this is a blueprint, where we may want to keep implementation details out of. However, choosing the right partitioning key is crucial to designing this, it shouldn't come as an aftermath.
With identifying the partitioning key, we would also define the partitioning strategy as a second step (hash- or range-based). This ultimately defines how we distribute data internally.
I think both defining the partitioning key as well as the strategy should be included in the blueprint. WDYT?
- Resolved by Grzegorz Bizon
- Resolved by Grzegorz Bizon
- Resolved by Grzegorz Bizon
mentioned in epic &4785 (closed)
mentioned in issue gitlab-com/www-gitlab-com#9518 (closed)
mentioned in epic &4685 (closed)
- Resolved by Grzegorz Bizon
- Resolved by Grzegorz Bizon
- Resolved by Grzegorz Bizon
- Resolved by Grzegorz Bizon
mentioned in issue #215646 (closed)
added 3873 commits
-
2765d004...2d42057d - 3872 commits from branch
master
- 9b6479fe - Improve CI/CD builds data storage model
-
2765d004...2d42057d - 3872 commits from branch
added 1 commit
- 819e2f88 - Add Jackie as a DRI in CI/CD data model architectural blueprint
mentioned in issue #271612 (closed)
added 1 commit
- e954a930 - Describe a proposal and add epics to ci_builds blueprint
@ayufan can you please review this blueprint as an Architecture Evolution Coach? Thanks in advance!
assigned to @ayufan
added 1 commit
- 7a4b8434 - Describe a proposal and add epics to ci_builds blueprint
- Resolved by Grzegorz Bizon
- Resolved by Grzegorz Bizon
added 1 commit
- 59e75f8f - Add a note about product changes related to ci_builds partitioning
mentioned in issue gitlab-com/www-gitlab-com#10808 (closed)
mentioned in issue gitlab-org/quality/triage-reports#2073 (closed)
mentioned in issue gitlab-org/quality/triage-reports#2207 (closed)
added wg_database-scalability label
mentioned in issue gitlab-com/www-gitlab-com#10568 (closed)