Skip to content
Snippets Groups Projects
Commit ff7c2b5a authored by Manoj M J's avatar Manoj M J :speech_balloon:
Browse files

Add next desired sharding keys for package_registry

Add "next" desired sharding keys for feature category `package_registry`.

These tables have been identified as a [cell local tables](https://docs.gitlab.com/ee/development/database/multiple_databases.html#guidelines-on-choosing-between-gitlab_main_cell-and-gitlab_main_clusterwide-schema).
All cell local tables require a [sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#defining-a-sharding-key-for-all-cell-local-tables)
or a [desired sharding key](https://docs.gitlab.com/ee/development/database/multiple_databases.html#defining-a-desired_sharding_key-for-automatically-backfilling-a-sharding_key)

A desired sharding key has been automatically selected for these tables.
These keys were chosen as the desired sharding keys, and we are labelling them as "next" because:

- this table does not have any belongs_to: relationship to a table that itself has a `NOT NULL` sharding key.
- however, this table has at least one belongs_to: relationship to a table that has a `desired sharding` key, that is yet to be backfilled.
- We work on the assumption that the desired sharding key on this parent association will be eventually backfilled, and when that is completed, the child table can inturn backfill from that parent association.

We use `awaiting_backfill_on_parent: true` to distinguish between "next" desired sharding keys and the normal desired sharding keys.

Additionally, `gitlab_schema` has been set to `gitlab_main_cell` for any tables didn't use this schema already.
For these tables we have also added `allow_cross_joins`, `allow_cross_transactions` and
`allow_cross_foreign_keys`. These will silence any existing violations, allowing the pipeline to pass without
requiring further changes. In the future, we'll remove these `allow_...` statements and fix any violations as
they arise. You can read more about this in the [documentation for multiple databases](https://docs.gitlab.com/ee/development/database/multiple_databases.html)

We have assigned a random backend engineer from ~"group::package registry" to review these changes.

Please confirm that:

- each of these tables can be classified as cell local
- the selected desired sharding key is appropriate
- the backfill configuration for the desired sharding key is correct

When you are finished, please request a review from the database maintainer suggested by Danger.
If you have any questions or concerns, reach out to @tigerwnz, @DylanGriffith or @manojmj.

You can see the POC MR of this approach [here](!145135)

This change was generated by
[gitlab-housekeeper](https://gitlab.com/gitlab-org/gitlab/-/tree/master/gems/gitlab-housekeeper)
using the Keeps::DetermineNextDesiredShardingKeyFeatureCategory keep.

To provide feedback on your experience with `gitlab-housekeeper` please comment in
<#442003>.

Changelog: other
parent 9948ec39
No related branches found
No related tags found
1 merge request!145254Add next desired sharding keys for package_registry
......@@ -7,4 +7,20 @@ feature_categories:
description: Conan package file metadata
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16418
milestone: '12.5'
gitlab_schema: gitlab_main
gitlab_schema: gitlab_main_cell
allow_cross_joins:
- gitlab_main_clusterwide
allow_cross_transactions:
- gitlab_main_clusterwide
allow_cross_foreign_keys:
- gitlab_main_clusterwide
desired_sharding_key:
project_id:
references: projects
backfill_via:
parent:
foreign_key: package_file_id
table: packages_package_files
sharding_key: project_id
belongs_to: package_file
awaiting_backfill_on_parent: true
......@@ -7,4 +7,20 @@ feature_categories:
description: Debian package file metadata
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49692
milestone: '13.8'
gitlab_schema: gitlab_main
gitlab_schema: gitlab_main_cell
allow_cross_joins:
- gitlab_main_clusterwide
allow_cross_transactions:
- gitlab_main_clusterwide
allow_cross_foreign_keys:
- gitlab_main_clusterwide
desired_sharding_key:
project_id:
references: projects
backfill_via:
parent:
foreign_key: package_file_id
table: packages_package_files
sharding_key: project_id
belongs_to: package_file
awaiting_backfill_on_parent: true
......@@ -7,4 +7,20 @@ feature_categories:
description: Debian project-level component files
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52885
milestone: '13.9'
gitlab_schema: gitlab_main
gitlab_schema: gitlab_main_cell
allow_cross_joins:
- gitlab_main_clusterwide
allow_cross_transactions:
- gitlab_main_clusterwide
allow_cross_foreign_keys:
- gitlab_main_clusterwide
desired_sharding_key:
project_id:
references: projects
backfill_via:
parent:
foreign_key: component_id
table: packages_debian_project_components
sharding_key: project_id
belongs_to: component
awaiting_backfill_on_parent: true
......@@ -7,4 +7,20 @@ feature_categories:
description: Helm package file metadata
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57017
milestone: '13.12'
gitlab_schema: gitlab_main
gitlab_schema: gitlab_main_cell
allow_cross_joins:
- gitlab_main_clusterwide
allow_cross_transactions:
- gitlab_main_clusterwide
allow_cross_foreign_keys:
- gitlab_main_clusterwide
desired_sharding_key:
project_id:
references: projects
backfill_via:
parent:
foreign_key: package_file_id
table: packages_package_files
sharding_key: project_id
belongs_to: package_file
awaiting_backfill_on_parent: true
......@@ -7,4 +7,20 @@ feature_categories:
description: Join table between nuget target frameworks and packages_dependency_links
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/30618
milestone: '13.0'
gitlab_schema: gitlab_main
gitlab_schema: gitlab_main_cell
allow_cross_joins:
- gitlab_main_clusterwide
allow_cross_transactions:
- gitlab_main_clusterwide
allow_cross_foreign_keys:
- gitlab_main_clusterwide
desired_sharding_key:
project_id:
references: projects
backfill_via:
parent:
foreign_key: dependency_link_id
table: packages_dependency_links
sharding_key: project_id
belongs_to: dependency_link
awaiting_backfill_on_parent: true
......@@ -7,4 +7,20 @@ feature_categories:
description: Join table relating packages_package_files and ci_pipelines
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44348
milestone: '13.6'
gitlab_schema: gitlab_main
gitlab_schema: gitlab_main_cell
allow_cross_joins:
- gitlab_main_clusterwide
allow_cross_transactions:
- gitlab_main_clusterwide
allow_cross_foreign_keys:
- gitlab_main_clusterwide
desired_sharding_key:
project_id:
references: projects
backfill_via:
parent:
foreign_key: package_file_id
table: packages_package_files
sharding_key: project_id
belongs_to: package_file
awaiting_backfill_on_parent: true
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment