Add sharding key tracking issues for geo_replication

Add sharding key tracking issues for feature category geo_replication.

Background

These tables were unable to be classified automatically, and will require manual input. Eventually all tables will need to be correctly classified, but we understand that this will be complex for some tables and completing these will take time. Instead, our goal for this task is to ensure all remaining tables are tracked in an issue, and to classify any straightforward cases that our automation may have missed (options 1 and 2 below).

We have assigned a random backend engineer from groupgeo as the initial DRI for this task, as well as an engineering manager for visibility. Please note that we are not requesting a large time commitment, creating one issue and linking it for all tables is perfectly acceptable.

When you are finished, please assign to the database reviewer/maintainer suggested by Danger.

If you have any questions or concerns, reach out to #g_tenant-scale.

Task

For each table, please select one of the following options:

Option 1: Add a sharding key tracking issue

This option is best suited to tables whose sharding behaviour is unknown, or will require additional work before a sharding key can be defined.

Replace the TODO in the dictionary file with a link to an issue in the gitlab-org/gitlab project.

- sharding_key_issue_url: TODO
+ sharding_key_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/1234

You can create a new issue or link an existing one, and multiple entries can refer to the same issue. These issues will be used to track the work remaining on the progress dashboard.

If you are creating a new issue, you can copy over the following contents to the issue description:

Click to expand

Issue Title: Set sharding keys for tables in 'group::geo'

Issue Description:

Sharding keys need to be set for the tables: geo_cache_invalidation_events, geo_event_log, geo_events, geo_node_statuses, geo_nodes, geo_repositories_changed_events, group_wiki_repository_states, merge_request_diff_details, upload_states

This involves choosing one of the following, based on the intended behaviour of the table:

• The table is not cell-local
  • Set gitlab_schema to gitlab_main_clusterwide.
• The table is cell-local and requires a sharding key
  • Set gitlab_schema to gitlab_main_cell
  • Add a sharding_key or desired_sharding_key configuration. If the configuration is known but the chosen key doesn't yet meet not-null and foreign key requirements, you can add an exception to allowed_to_be_missing_not_null or allowed_to_be_missing_foreign_key to get the pipeline passing. Please link to a follow-up issue in a code comment next to the exception.
  • You may also need to set allow_cross_joins, allow_cross_transactions and allow_cross_foreign_keys if changing the schema causes pipeline failures. See db/docs/epics.yml for an example.
• The table is cell-local and does not require a sharding key
  • Set gitlab_schema to gitlab_main_cell and
  • Set exempt_from_sharding to true.

Documentation

• Choosing either the gitlab_main_cell or gitlab_main_clusterwide schema
• Defining a sharding key for all cell-local tables
• Defining a desired_sharding_key to automatically backfill a sharding_key

Option 2: Add sharding key configuration

This option is best suited to tables with an easily identifiable sharding key that will require minimal work to define.

Remove sharding_key_issue_url from the dictionary file and instead complete the classification for the table. This involves choosing one of the following, based on the intended behaviour of the table:

• The table is not cell-local
  • Set gitlab_schema to gitlab_main_clusterwide.
• The table is cell-local and requires a sharding key
  • Set gitlab_schema to gitlab_main_cell
  • Add a sharding_key or desired_sharding_key configuration. If the configuration is known but the chosen key doesn't yet meet not-null and foreign key requirements, you can add an exception to allowed_to_be_missing_not_null or allowed_to_be_missing_foreign_key to get the pipeline passing. Please link to a follow-up issue in a code comment next to the exception.
  • You may also need to set allow_cross_joins, allow_cross_transactions and allow_cross_foreign_keys if changing the schema causes pipeline failures. See db/docs/epics.yml for an example.
• The table is cell-local and does not require a sharding key
  • Set gitlab_schema to gitlab_main_cell and
  • Set exempt_from_sharding to true.

Documentation

• Choosing either the gitlab_main_cell or gitlab_main_clusterwide schema
• Defining a sharding key for all cell-local tables
• Defining a desired_sharding_key to automatically backfill a sharding_key

Related to #455137 (closed)

This change was generated by gitlab-housekeeper using the Keeps::AddShardingKeyTrackingIssues keep.

To provide feedback on your experience with gitlab-housekeeper please comment in #442003 (closed).

added automation:gitlab-housekeeper-authored database groupgeo maintenancescalability + 1 deleted label

restored source branch add-sharding-key-tracking-issues-geo_replication

added typefeature label

removed maintenancescalability label

added devopssystems sectioncore platform labels

added maintenancescalability label

added 1 commit

• 141d5f68 - Draft: Add sharding key tracking issues for geo_replication

Compare with previous version

added typemaintenance label and removed typefeature label

added pipelinetier-1 label and removed 1 deleted label

added automation:bot-authored label

added 1006 commits

• 141d5f68...aedea43e - 1005 commits from branch master
• 4c000f32 - Draft: Add sharding key tracking issues for geo_replication

Compare with previous version

added databasereview pending label

	3 Warnings
	4f4d5b56: The commit body should not contain more than 72 characters per line. For more information, take a look at our Commit message guidelines.
	b4949aa4: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
	The master pipeline status page reported failures in rspec system pg14 single-db-ci-connection 22/32 rspec system pg14 single-db 22/32 rspec system pg14 22/32 If these jobs fail in your merge request with the same errors, then they are not caused by your changes. Please check for any on-going incidents in the incident issue tracker or in the #master-broken Slack channel.

Reviewer roulette

Category	Reviewer	Maintainer
database	@fcatteau (UTC+2)	@DylanGriffith (UTC+10)

Please check reviewer's status!

• Reviewer is available!
• Reviewer is unavailable!

Please refer to documentation page for guidance on how you can benefit from the Reviewer Roulette, or use the GitLab Review Workload Dashboard to find other available reviewers.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

assigned to @mkozono and @juan-silva

unassigned @juan-silva

assigned to @pjphillips

@mkozono Friendly ping about this.

changed milestone to %17.1

@dbalexandre Could you double check my proposals?

@arturoherrero Is there a discussion about the uploads table already?

added 1 commit

• 23db4f77 - Define schema and sharding keys for Geo tables

Compare with previous version

added 1 commit

• 09e30118 - Add desired sharding keys to upload_states

Compare with previous version

added 1 commit

• 38b8ac0c - Use easiest schema since this table will be removed

Compare with previous version

resolved all threads

added 1 commit

• 433b82ce - Fix group_wiki_repository_states db doc

Compare with previous version

added 1 commit

• b1c2fa18 - Remove leftover line

Compare with previous version

added 2439 commits

• b1c2fa18...9919fce0 - 2437 commits from branch master
• 4f4d5b56 - Draft: Add sharding key tracking issues for geo_replication
• b4949aa4 - Define schema and sharding keys for Geo tables

Compare with previous version

@fcatteau Would you please review?

requested review from @fcatteau

mentioned in issue #464440 (closed)

marked this merge request as ready

added databasereviewed label and removed databasereview pending label

approved this merge request

requested review from @DylanGriffith and removed review request for @fcatteau

added pipeline:mr-approved label

added pipelinetier-2 label and removed pipelinetier-1 label

@fcatteau, thanks for approving this merge request.

This is the first time the merge request has been approved. To ensure we don't only run predictive pipelines, and we don't break master, please start a new pipeline before merging.

Please wait for the pipeline to start before resolving this discussion and set auto-merge for the new pipeline. See merging a merge request for more details.

added 1 commit

• 6357a601 - Exempt geo_repositories_changed_events from sharding

Compare with previous version

added 1 commit

• 7cb9b36a - Fix geo_repositories_changed_events schema

Compare with previous version