Tracking Sharding Key Backfill Progress
This is the tracker issue for backfilling and creating sharding key for tables.
Dashboard URL: https://cells-progress-tracker-gitlab-org-tenant-scale-g-f4ad96bf01d25f.gitlab.io/sharding_keys
Tables for backfill and creating sharding key
| Table | MR Link | Backfill Status | Finalize Status | NOT NULL Status |
|---|---|---|---|---|
| ml_candidate_params | !165336 (merged) | |||
| ml_candidate_metrics | !164465 (merged) | |||
| alert_management_alert_assignees | !165673 (merged) | |||
| ci_job_artifact_states | !165940 (merged) | |||
| alert_management_alert_metric_images | !166074 (merged) | |||
| incident_management_escalation_rules | !166149 (merged) | |||
| incident_management_oncall_rotations | !166153 (merged) !184622 (merged) | |||
| alert_management_alert_user_mentions | !166145 (merged) | |||
| milestone_releases | !166466 (merged) | |||
| incident_management_timeline_event_tag_links | !166461 (merged) | |||
| incident_management_pending_alert_escalations | !166816 (merged) | |||
| ci_unit_test_failures | !166817 (merged) | |||
| packages_composer_metadata | !167444 (merged) | |||
| packages_npm_metadata | !167446 (merged) | |||
| packages_nuget_metadata | !167607 (merged) | |||
| packages_pypi_metadata | !167608 (merged) | |||
| packages_rpm_metadata | !167609 (merged) | |||
| packages_rubygems_metadata | !167611 (merged) | |||
| pages_deployment_states | !167613 (merged) | |||
| pages_domain_acme_orders | !167614 (merged) | |||
| ci_secure_file_states | !167617 (merged) | |||
| dast_scanner_profiles_builds | !167626 (merged) | |||
| dast_profiles_pipelines | !167633 (merged) | |||
| ci_builds_runner_session | !167634 (merged) | |||
| ci_resources | !167645 (merged) | |||
| approval_project_rules_protected_branches | !167648 (merged) | |||
| dast_site_profiles_builds | !167768 (merged) | |||
| ci_pipeline_schedule_variables | !168110 (merged) | |||
| deployment_merge_requests | !183738 (merged) | |||
| security_findings | !169121 (merged) | |||
| dependency_proxy_blob_states | !169237 (merged) | |||
| dependency_proxy_manifest_states | !169239 (merged) | |||
| container_repository_states | !169240 (merged) | |||
| group_wiki_repository_states | !169242 (merged) | |||
| analytics_cycle_analytics_value_stream | !169585 (merged) | |||
| issuable_severities | !173728 (merged) | |||
| issue_assignees | !174106 (merged) | |||
| issue_user_mentions | !174399 (merged) | |||
| resource_link_events | !174402 (merged) | |||
| resource_weight_events | !174405 (merged) | |||
| work_item_progresses | !174410 (merged) | |||
| incident_management_issuable_escalations | !174415 (merged) | |||
| issuable_resource_links | !174419 (merged) | |||
| issue_assignment_events | !174500 (merged) | |||
| issue_metrics | !174502 (merged) | |||
| issue_customer_relations_contacts | !174504 (merged) | |||
| issue_email_participants | !174506 (merged) | |||
| issue_emails | !174507 (merged) | |||
| issuable_metric_images | !174508 (merged) | |||
| sentry_issues | !174555 (merged) | |||
| protected_branch_merge_access_levels | !174564 (merged) | |||
| status_page_published_incidents | !174868 (merged) | |||
| snippet_repositories | !175410 (merged) | |||
| protected_branch_push_access_levels | !175714 (merged) | |||
| protected_branch_unprotect_access_levels | !175718 (merged) | |||
| issuable_slas | !175719 (merged) | |||
| bulk_import_export_uploads | !175723 (merged) | |||
| boards_epic_list_user_preferences | !180037 (merged) | |||
| dast_pre_scan_verification_steps | !180046 (merged) | |||
| design_management_designs_versions | !180220 (merged) | |||
| operations_scopes | !180223 (merged) | |||
| design_management_repository_states | !180426 (merged) | |||
| merge_request_diff_details | !180428 (merged) | |||
| bulk_import_export_batches | !180432 (merged) | |||
| bulk_import_failures | !180435 (merged) | |||
| bulk_import_trackers | !180437 (merged), !184776 (merged) | |||
| required_code_owners_sections | !180440 (merged) | |||
| snippet_repository_storage_moves | !180441 (merged) | |||
| snippet_user_mentions | !180836 (merged) | |||
| design_user_mentions | !180837 (merged) | |||
| requirements_management_test_reports | !180838 (merged) | |||
| merge_request_predictions | !180839 (merged) | |||
| merge_request_cleanup_schedules | !180840 (merged) | |||
| subscription_user_add_on_assignments | !169834 (merged) !170136 (merged) | |||
| security_scans | !171458 (merged) !171845 (merged) | |||
| dependency_list_exports | !171854 (merged) !172178 (merged) | |||
| agent_activity_events | !170139 (merged) | |||
| approval_group_rules_users | !169836 (merged) | |||
| project_relation_export_uploads | !182116 (merged) | |||
| incident_management_pending_issue_escalations | !183047 (merged) | |||
| push_event_payloads | !183123 (merged) | |||
| snippet_statistics | !183124 (merged) | |||
| approval_merge_request_rules_approved_approvers | !183352 (merged) | |||
| approval_merge_request_rules_users | !183354 (merged) | |||
| wiki_page_slugs | !183332 (merged) | |||
| merge_request_context_commit_diff_files | !183935 (merged) | |||
| packages_nuget_dependency_link_metadata | !184610 (merged) | |||
| incident_management_oncall_participants | !184771 (merged) | |||
| incident_management_oncall_shifts | !184772 (merged) | |||
| packages_package_files | !184630 (merged) | |||
| snippet_repositories | !175410 (merged) | |||
| packages_package_file_build_infos | !185226 (merged) | |||
| packages_helm_file_metadata | !185225 (merged) | |||
| packages_debian_file_metadata | !185223 (merged) | |||
| packages_conan_file_metadata | !185222 (merged) | |||
| bulk_import_configurations | !187337 (merged) |
Tables needed to complete first
Current Sharding key status for the tables being marked as exempt from shard. Once it's at last stage please remove the sharding_key_issue_url from the db/docs:
| Table | MR Link | Sharding Key | Backfill Status | Finalize Status | NOT NULL Status | Loose FK Status |
|---|---|---|---|---|---|---|
| ci_job_artifact_states | !165940 (merged) | |||||
| ci_secure_file_states | !167617 (merged) | |||||
| dependency_proxy_blob_states | !169237 (merged) | |||||
| dependency_proxy_manifest_states | !169239 (merged) | |||||
| group_wiki_repositories | Swap FK to LFK | |||||
| group_wiki_repository_states | !169242 (merged) | |||||
| lfs_objects | ||||||
| lfs_object_states | ||||||
| merge_request_diff_details | !180428 (merged) | |||||
| pages_deployment_states | !167613 (merged) | |||||
| pool_repositories | ||||||
| project_repositories | Swap FK to LFK | |||||
| project_states | Swap FK to LFK | |||||
| snippet_repositories | !175410 (merged) | |||||
| snippet_repository_states | ||||||
| upload_states |
Tables for NOT NULL constraint
| Table | Merge Request | Constraint added? | Validation finalized? |
|---|---|---|---|
| release_links | !177235 (merged) | ||
| cluster_agent_tokens | !177246 (merged) | ||
| ml_experiment_metadata | !177249 (merged) | ||
| alert_management_alert_assignees | !177252 (merged) | ||
| alert_management_alert_metric_images | !177257 (merged) | ||
| alert_management_alert_user_mentions | !177258 (merged) | ||
| boards_epic_board_labels | !177343 (merged) | ||
| boards_epic_board_positions | !177347 (merged) | ||
| boards_epic_lists | !177348 (merged) | ||
| boards_epic_user_preferences | !177349 (merged) | ||
| related_epic_links | !177352 (merged) | ||
| dora_daily_metrics | !177353 (merged) | ||
| dast_pre_scan_verifications | !177354 (merged) | ||
| dast_profiles_pipelines | !177355 (merged) | ||
| dast_profiles_tags | !177357 (merged) | ||
| dast_scanner_profiles_builds | !177358 (merged) | ||
| dast_site_profile_secret_variables | !177359 (merged) | ||
| dast_site_validations | !177360 (merged) | ||
| sbom_occurrences_vulnerabilities | !177361 (merged), !179798 (merged) | ||
| packages_build_infos | !177362 (merged), !182526 (merged) | ||
| packages_debian_group_distribution_keys | !177363 (merged) | ||
| packages_debian_project_distribution_keys | !177365 (merged) | ||
| packages_debian_publications | !177366 (merged) | ||
| packages_tags | constraint already exists | ||
| design_management_designs | !177367 (merged) | ||
| design_management_repositories | !177368 (merged) | ||
| operations_feature_flags_issues | !177370 (merged) | ||
| operations_strategies | !177371 (merged) | ||
| operations_strategies_user_lists | !177372 (merged) | ||
| vulnerability_finding_evidences | !177374 (merged), !179802 (merged) | ||
| vulnerability_finding_links | !177375 (merged), !180689 (merged) | ||
| vulnerability_finding_signatures | !177376 (merged), !180692 (merged) | ||
| vulnerability_findings_remediations | !177377 (merged), !180698 (merged) | ||
| vulnerability_flags | !177380 (merged) | ||
| vulnerability_user_mentions | !177381 (merged) | ||
| workspace_variables | !177383 (merged) | ||
| terraform_state_versions | !177384 (merged), !180698 (merged) | ||
| audit_events_streaming_event_type_filters | !177385 (merged) | ||
| audit_events_streaming_headers | !177386 (merged) | ||
| compliance_framework_security_policies | !177388 (merged) | ||
| wiki_repository_states | !177390 (merged), !180700 (merged) | ||
| user_achievements | !177391 (merged) | ||
| status_check_responses | !177392 (merged), !180701 (merged) | ||
| approval_group_rules_protected_branches | !177396 (merged) | ||
| approval_project_rules_users | !177398 (merged), !180908 (merged) | ||
| merge_request_blocks | !177399 (merged), !180913 (merged) | ||
| approval_merge_request_rule_sources | !177400 (merged), !180906 (merged) | ||
| approval_merge_request_rules | !177401 (merged), !180907 (merged) | ||
| approvals | !177402 (merged), !181753 (merged) | ||
| draft_notes | !177403 (merged), !180909 (merged) | ||
| merge_request_assignees | !177404 (merged), !180912 (merged) | ||
| merge_request_context_commits | !177405 (merged) | ||
| merge_request_diffs | constraint already exists | ||
| project_relation_exports | !177407 (merged), !180914 (merged) | ||
| epic_user_mentions | !177411 (merged), !180910 (merged) | ||
| issue_links | !177412 (merged), !180911 (merged) | ||
| work_item_parent_links | !177413 (merged), !180915 (merged) | ||
| ml_candidate_metadata | !179814 (merged) | ||
| ml_model_metadata | !179816 (merged) | ||
| external_status_checks_protected_branches | !181060 (merged) | ||
| packages_conan_metadata | !181061 (merged) | ||
| packages_debian_group_architectures | !181062 (merged) | ||
| packages_debian_group_components | !181064 (merged) | ||
| packages_debian_project_architectures | !181065 (merged) | ||
| vulnerability_issue_links | !181079 (merged) | ||
| vulnerability_merge_request_links | !181080 (merged) | ||
| merge_request_reviewers | !181081 (merged), !182524 (merged) | ||
| merge_requests_closing_issues | !181082 (merged) | ||
| ci_unit_test_failures | !182507 (merged) | ||
| packages_dependency_links | !182510 (merged) | ||
| packages_debian_project_components | !182509 (merged) | ||
| packages_maven_metadata | !182511 (merged) | ||
| evidences | !182508 (merged) | ||
| ci_pipeline_schedule_variables | !182681 (merged) | ||
| design_management_versions | !177369 (merged) | ||
| ci_resources | !182693 (merged) | ||
| merge_request_users_mentions | !182696 (merged) | ||
| vulnerability_external_issue_links | !182697 (merged) | ||
| vulnerability_state_transitions | !182698 (merged) | ||
| ci_trigger_requests | !183978 (merged) | ||
| ci_build_pending_states | !183975 (merged) | ||
| ci_build_trace_chunks | !183976 (merged) | ||
| ci_builds_runner_session | !183977 (merged) | ||
| epic_issues | !184430 (merged) | ||
| ci_build_needs | !184429 (merged) | ||
| protected_environment_approval_rules | !184433 (merged) | ||
| ml_candidate_params | !184432 (merged) | ||
| ml_candidate_metrics | !184431 (merged) | ||
| protected_environment_deploy_access_levels | !184913 (merged) | ||
| error_tracking_error_events | !184908 (merged) | ||
| vulnerability_occurrence_identifiers | !184914 (merged) |
Implementation Hacks
- Get table information using following from the console:
gdk psql
\d <table_name>1. Tables containing non :id primary key
- Replace
:idwith non-id primary key here, here and here. - Comment this line here.
- After MR is generated, override
batch_columnby adding it inbackfil_table_name_project_id_spec.rblike we did it here.
Example MR: !165940 (merged)
if the table uses
gitlab_cidb then we need to specifymigration: :gitlab_ciin migration files.
2. Tables containing composite primary keys
Example MR: !166466 (merged)
3. Tables containing partitioned
4. Tables containing no backend engineers in group data
5. Tables containing issues.namespace_id backfilling
6. Tables in ci database and using project_id as sharding_key
Edited by 🤖 GitLab Bot 🤖