Skip to content

Backfill data from merge requests table into compliance violations table

What does this MR do and why?

  1. NOTE: This MR is currently opened against !107998 (merged) and I'll switch it to master once that MR is merged.
  2. Creates a background migration to copy the following data
    • Copy title, target_branch and target_project_id from merge_requests table into merge_requests_compliance_violations table.
    • Copy merged_at column from merge_request_metrics table into merge_requests_compliance_violations table.
  3. This is an irreversible migration, however, in case we find an issue with the migrated data after deploying on production, we can write another migration and override these columns again. The new columns defined merge_requests_compliance_violations table in !107998 (merged) are not used in the application code as of now.
  4. Affecting 3215346 rows.
gitlabhq_dblab=# select count(*) from merge_requests_compliance_violations;
  count
---------
 3215346
(1 row)

Database queries

  1. https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/15679/commands/54271
UPDATE
    "merge_requests_compliance_violations"
SET
    "merged_at" = '2022-10-27 06:11:43.694453',
    "target_project_id" = 17,
    "title" = 'Ab sunt fuga ut voluptatum atque voluptatem.',
    "target_branch" = 'main'
WHERE
    "merge_requests_compliance_violations"."id" = 648
  1. https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/15679/commands/54267
SELECT
    "merge_requests_compliance_violations".*
FROM
    "merge_requests_compliance_violations"
WHERE
    "merge_requests_compliance_violations"."id" >= 1
    AND "merge_requests_compliance_violations"."id" < 500

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

image

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Before running the migration ensure that you have a few rows in merge_requests_compliance_violations_table by running the following in rails console
MergeRequests::ComplianceViolation.count
  1. In case you don't have any rows you can seed the data by running the following in your shell
FILTER=compliance_report_violations rake db:seed_fu
  1. Verify that title, target_project_id, target_branch and merged_at columns in the merge_requests_compliance_violations table is nil. If not you can forcefully make them nil by running the following in rails console
MergeRequests::ComplianceViolation.update_all(:title => nil)
  1. Check the count of rows with nil title by running
MergeRequests::ComplianceViolation.where(:title => nil).count
  1. Queue background migration by running bundle exec rails db:migrate
  2. Verify that the batched background migration BackfillComplianceViolations is created by running the following in rails console.
Gitlab::Database::BackgroundMigration::BatchedMigration.find_by_job_class_name('BackfillComplianceViolations')
  1. Visit the background_migrations admin dashboard and verify that the background migration is completed successfully.
  2. Ensure that we have no rows with nil values for any of the above columns. You can again check the count by running the following in rails console
MergeRequests::ComplianceViolation.where(:title => nil).count # This will check for nil values for the `title` column
  1. Ensure manually that the title is populated with the correct data i.e. the MR's title for which the compliance violation record is created. Repeat this step for other columns as well.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #385500 (closed)

Edited by Huzaifa Iftikhar

Merge request reports

Loading