Skip to content
Snippets Groups Projects

GitLab CI/CD scalability improvements blueprint

Merged Grzegorz Bizon requested to merge architecture/gb/improve-ci-builds-data-model into master

What does this MR do?

This merge request adds to docs an architectural blueprint about improving GitLab CI/CD scalability and CI/CD data storage model.

See https://gitlab.com/gitlab-org/architecture/tasks/-/issues/5

Edited by Grzegorz Bizon

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Grzegorz Bizon added 1 commit

    added 1 commit

    • 10c6bed3 - Improve CI/CD builds data storage model

    Compare with previous version

  • Grzegorz Bizon added 1 commit

    added 1 commit

    • 913bf6d1 - Improve CI/CD builds data storage model

    Compare with previous version

  • Grzegorz Bizon added 1 commit

    added 1 commit

    • 8b132927 - Improve CI/CD builds data storage model

    Compare with previous version

    • Resolved by Grzegorz Bizon

      @grzesiek

      Typically, the main and first question we'd need to answer with partitioning is - what partitioning key do we choose and what is the trade-off (how does it help us scale, what are the drawbacks)?

      Previously, we've dug into the application behavior and identified 'access patterns' - i.e. what keys do we typically use to access data. An ideal partitioning key is always present when we query data. If that is not the case, we're likely to end up with a trade off having a "good" way of accessing data (using the partitioning key) and a "much worse" way (crossing partitions).

      I understand this is a blueprint, where we may want to keep implementation details out of. However, choosing the right partitioning key is crucial to designing this, it shouldn't come as an aftermath.

      With identifying the partitioning key, we would also define the partitioning strategy as a second step (hash- or range-based). This ultimately defines how we distribute data internally.

      I think both defining the partitioning key as well as the strategy should be included in the blueprint. WDYT?

  • mentioned in epic &4785 (closed)

  • mentioned in epic &4685 (closed)

  • Grzegorz Bizon added 1 commit

    added 1 commit

    • 1cb13658 - Improve CI/CD builds data storage model

    Compare with previous version

  • Grzegorz Bizon added 1 commit

    added 1 commit

    • 2765d004 - Improve CI/CD builds data storage model

    Compare with previous version

  • mentioned in issue #215646 (closed)

  • Grzegorz Bizon added 3873 commits

    added 3873 commits

    Compare with previous version

  • Grzegorz Bizon added 1 commit

    added 1 commit

    • e1093460 - Improve CI/CD builds data storage model

    Compare with previous version

  • Grzegorz Bizon added 1 commit

    added 1 commit

    • 819e2f88 - Add Jackie as a DRI in CI/CD data model architectural blueprint

    Compare with previous version

  • mentioned in issue #271612 (closed)

  • Grzegorz Bizon added 1 commit

    added 1 commit

    • e954a930 - Describe a proposal and add epics to ci_builds blueprint

    Compare with previous version

  • Grzegorz Bizon marked this merge request as ready

    marked this merge request as ready

  • Grzegorz Bizon resolved all threads

    resolved all threads

  • Author Maintainer

    @ayufan can you please review this blueprint as an Architecture Evolution Coach? Thanks in advance! :purple_heart:

  • assigned to @ayufan

  • Grzegorz Bizon added 1 commit

    added 1 commit

    • 7a4b8434 - Describe a proposal and add epics to ci_builds blueprint

    Compare with previous version

  • Grzegorz Bizon added 1 commit

    added 1 commit

    • 59e75f8f - Add a note about product changes related to ci_builds partitioning

    Compare with previous version

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading