Skip to content

Add sharding key for events table

Sharding keys need to be set for the tables:

  • events

This involves choosing one of the following, based on the intended behaviour of the table:

  • The table is not cell-local
    • Set gitlab_schema to gitlab_main_clusterwide.
  • The table is cell-local and requires a sharding key
    • Set gitlab_schema to gitlab_main_cell
    • Add a sharding_key or desired_sharding_key configuration. If the configuration is known but the chosen key doesn't yet meet not-null and foreign key requirements, you can add an exception to allowed_to_be_missing_not_null or allowed_to_be_missing_foreign_key to get the pipeline passing. Please link to a follow-up issue in a code comment next to the exception.
    • You may also need to set allow_cross_joins, allow_cross_transactions and allow_cross_foreign_keys if changing the schema causes pipeline failures. See db/docs/epics.yml for an example.
  • The table is cell-local and does not require a sharding key
    • Set gitlab_schema to gitlab_main_cell and
    • Set exempt_from_sharding to true.

Documentation

Proposal

Based on #462801 (comment 2015672699).

  • %17.4:
    • Remove index_events_on_group_id_partial as it is a redundant to index_events_on_group_id_and_id and index_events_for_group_activity. (!164709 (merged))
    • Add column personal_namespace_id bigint:
    • Create index on (personal_namespace_id) WHERE personal_namespace_id IS NOT NULL:
    • Add FK (personal_namespace_id) REFERENCES namespaces(id) ON DELETE CASCADE:
  • %17.5 (required stop):
    • Update app to populate personal_namespace_id when no project_id or group_id is present. Create NOT VALID check constraint to ensure that at least one of project_id, group_id, and personal_namespace_id is not null (!164979 (merged))
  • %17.6:
    • Back-fill records where project_id is null and group_id is null. (!166563 (merged))
  • %17.9:
    • Finalize the backfill background migration. (!177810 (merged))
    • Validate the check_events_sharding_key_is_not_null constraint. (!177810 (merged))
    • Define composite sharding key on project_id, group_id, and personal_namespace_id. (!177810 (merged))
Edited by Krasimir Angelov