Add sharding key for events table
Sharding keys need to be set for the tables:
- events
This involves choosing one of the following, based on the intended behaviour of the table:
-
The table is not cell-local
- Set
gitlab_schematogitlab_main_clusterwide.
- Set
-
The table is cell-local and requires a sharding key
- Set
gitlab_schematogitlab_main_cell - Add a
sharding_keyordesired_sharding_keyconfiguration. If the configuration is known but the chosen key doesn't yet meet not-null and foreign key requirements, you can add an exception toallowed_to_be_missing_not_nullorallowed_to_be_missing_foreign_keyto get the pipeline passing. Please link to a follow-up issue in a code comment next to the exception. - You may also need to set
allow_cross_joins,allow_cross_transactionsandallow_cross_foreign_keysif changing the schema causes pipeline failures. Seedb/docs/epics.ymlfor an example.
- Set
-
The table is cell-local and does not require a sharding key
- Set
gitlab_schematogitlab_main_celland - Set
exempt_from_shardingtotrue.
- Set
Documentation
- Choosing either the gitlab_main_cell or gitlab_main_clusterwide schema
- Defining a sharding key for all cell-local tables
- Defining a desired_sharding_key to automatically backfill a sharding_key
Proposal
Based on #462801 (comment 2015672699).
-
%17.4:
-
Remove index_events_on_group_id_partialas it is a redundant toindex_events_on_group_id_and_idandindex_events_for_group_activity. (!164709 (merged)) -
Add column personal_namespace_id bigint:-
GitLab.com (!165127 (merged)) -
Self-managed (!165591 (merged))
-
-
Create index on (personal_namespace_id) WHERE personal_namespace_id IS NOT NULL:-
GitLab.com -
async (!165127 (merged)) - [-]
sync(not needed, done in !165591 (merged))
-
-
Self-managed (!165591 (merged))
-
-
Add FK (personal_namespace_id) REFERENCES namespaces(id) ON DELETE CASCADE:-
GitLab.com -
Self-managed (!165591 (merged))
-
-
-
%17.5 (required stop):
-
Update app to populate personal_namespace_idwhen noproject_idorgroup_idis present. Create NOT VALID check constraint to ensure that at least one ofproject_id,group_id, andpersonal_namespace_idis not null (!164979 (merged))
-
-
%17.6:
-
Back-fill records where project_id is null and group_id is null. (!166563 (merged))
-
-
%17.9:
-
Finalize the backfill background migration. (!177810 (merged)) -
Validate the check_events_sharding_key_is_not_nullconstraint. (!177810 (merged)) -
Define composite sharding key on project_id,group_id, andpersonal_namespace_id. (!177810 (merged))
-
Edited by Krasimir Angelov