Skip to content

Add --filter-identifiers option to gitlab-housekeeper

Dylan Griffith requested to merge housekeeper-add-filter-identifiers-option into master

What does this MR do and why?

During local usage of this we found it is often desirable to get the housekeeper to generate a specific MR and ignore the other ones. Since it just does them in order we had to previously manually hack the keeps to skip the changes we don't care about.

This change introduces a command line option --filter-identifiers which takes a comma separated list which will be matched against the list of "identifiers" in a change yielded from a keep. The identifiers is an array so to allow for flexible regex matching of specific elements in the array we filter such that all regexes must match at least one of the identifiers in the list.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Output from running with a filter
$ bundle exec gitlab-housekeeper -m2 -d -r keeps/overdue_finalize_background_migration.rb -k Keeps::OverdueFinalizeBackgroundMigration --filter-identifiers 'Backfill.*Reads'
      create  db/post_migrate/20240130032730_finalize_backfill_finding_id_in_vulnerabilities.rb
Skipping change: ["OverdueFinalizeBackgroundMigration", "BackfillFindingIdInVulnerabilities"] due to not matching filter
      create  db/post_migrate/20240130032740_finalize_backfill_has_merge_request_of_vulnerability_reads.rb
=> OverdueFinalizeBackgroundMigration: BackfillHasMergeRequestOfVulnerabilityReads
=> Title:
Finalize migration BackfillHasMergeRequestOfVulnerabilityReads

=> Description:
This migration was finished at `2023-09-19 05:04:03 UTC`, you can confirm
the status using our
[batched background migration chatops commands](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#monitor-the-progress-and-status-of-a-batched-background-migration).
  To confirm it is finished you can run:


/chatops run batched_background_migrations status 1000456


The last time this background migration was triggered was in [db/post_migrate/20230907155247_queue_backfill_has_merge_request_of_vulnerability_reads.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/post_migrate/20230907155247_queue_backfill_has_merge_request_of_vulnerability_reads.rb)

  You can read more about the process for finalizing batched background migrations in
https://docs.gitlab.com/ee/development/database/batched_background_migrations.html .

  As part of our process we want to ensure all batched background migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can finalize any batched background migration that was added before the
last required stop.

=> Attributes:
Labels: automation:gitlab-housekeeper-authored

=> Diff:
diff --git a/db/docs/batched_background_migrations/backfill_has_merge_request_of_vulnerability_reads.yml b/db/docs/batched_background_migrations/backfill_has_merge_request_of_vulnerability_reads.yml
index d6cd709ca70a..a2190506fca0 100644
--- a/db/docs/batched_background_migrations/backfill_has_merge_request_of_vulnerability_reads.yml
+++ b/db/docs/batched_background_migrations/backfill_has_merge_request_of_vulnerability_reads.yml
@@ -4,3 +4,4 @@ description: Backfills has_merge_request column for vulnerability_reads table.
 feature_category: database
 introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130952
 milestone: '16.4'
+finalized_by: '20240130032740'
diff --git a/db/post_migrate/20240130032740_finalize_backfill_has_merge_request_of_vulnerability_reads.rb b/db/post_migrate/20240130032740_finalize_backfill_has_merge_request_of_vulnerability_reads.rb
new file mode 100644
index 000000000000..c76125bd14d2
--- /dev/null
+++ b/db/post_migrate/20240130032740_finalize_backfill_has_merge_request_of_vulnerability_reads.rb
@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+class FinalizeBackfillHasMergeRequestOfVulnerabilityReads < Gitlab::Database::Migration[2.2]
+  milestone '16.9'
+
+  disable_ddl_transaction!
+
+  restrict_gitlab_migration gitlab_schema: :gitlab_main
+
+  def up
+    ensure_batched_background_migration_is_finished(
+      job_class_name: 'BackfillHasMergeRequestOfVulnerabilityReads',
+      table_name: :vulnerability_reads,
+      column_name: :vulnerability_id,
+      job_arguments: [],
+      finalize: true
+    )
+  end
+
+  def down; end
+end
diff --git a/db/schema_migrations/20240130032740 b/db/schema_migrations/20240130032740
new file mode 100644
index 000000000000..8dc2e01adab1
--- /dev/null
+++ b/db/schema_migrations/20240130032740
@@ -0,0 +1 @@
+708890d332b90cb5be8b6232fe7c1f9ad0ede42682ca7f2a567eb939bde61682
\ No newline at end of file

      create  db/post_migrate/20240130032749_finalize_backfill_has_remediations_of_vulnerability_reads.rb
=> OverdueFinalizeBackgroundMigration: BackfillHasRemediationsOfVulnerabilityReads
=> Title:
Finalize migration BackfillHasRemediationsOfVulnerabilityReads

=> Description:
This migration was finished at `2024-01-25 11:23:05 UTC`, you can confirm
the status using our
[batched background migration chatops commands](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#monitor-the-progress-and-status-of-a-batched-background-migration).
  To confirm it is finished you can run:


/chatops run batched_background_migrations status 1000469

The last time this background migration was triggered was in [db/post_migrate/20231201204712_requeue2_backfill_has_remediations_of_vulnerability_reads.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/post_migrate/20231201204712_requeue2_backfill_has_remediations_of_vulnerability_reads.rb)

  You can read more about the process for finalizing batched background migrations in
https://docs.gitlab.com/ee/development/database/batched_background_migrations.html .

  As part of our process we want to ensure all batched background migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can finalize any batched background migration that was added before the
last required stop.

=> Attributes:
Labels: automation:gitlab-housekeeper-authored

=> Diff:
diff --git a/db/docs/batched_background_migrations/backfill_has_remediations_of_vulnerability_reads.yml b/db/docs/batched_background_migrations/backfill_has_remediations_of_vulnerability_reads.yml
index 18dd61379214..327b913ce837 100644
--- a/db/docs/batched_background_migrations/backfill_has_remediations_of_vulnerability_reads.yml
+++ b/db/docs/batched_background_migrations/backfill_has_remediations_of_vulnerability_reads.yml
@@ -1,9 +1,10 @@
 ---
 migration_job_name: BackfillHasRemediationsOfVulnerabilityReads
-description: Backfills has_remediations column for vulnerability_reads table.
-  Originally introduced via https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133714
-  RE-ran because there was a error in remediation ingestion logic.
+description: Backfills has_remediations column for vulnerability_reads table. Originally
+  introduced via https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133714 RE-ran
+  because there was a error in remediation ingestion logic.
 feature_category: database
 introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133714
 milestone: '16.7'
 queued_migration_version: 20231201204712
+finalized_by: '20240130032749'
diff --git a/db/post_migrate/20240130032749_finalize_backfill_has_remediations_of_vulnerability_reads.rb b/db/post_migrate/20240130032749_finalize_backfill_has_remediations_of_vulnerability_reads.rb
new file mode 100644
index 000000000000..25dad5ecabe1
--- /dev/null
+++ b/db/post_migrate/20240130032749_finalize_backfill_has_remediations_of_vulnerability_reads.rb
@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+class FinalizeBackfillHasRemediationsOfVulnerabilityReads < Gitlab::Database::Migration[2.2]
+  milestone '16.9'
+
+  disable_ddl_transaction!
+
+  restrict_gitlab_migration gitlab_schema: :gitlab_main
+
+  def up
+    ensure_batched_background_migration_is_finished(
+      job_class_name: 'BackfillHasRemediationsOfVulnerabilityReads',
+      table_name: :vulnerability_reads,
+      column_name: :vulnerability_id,
+      job_arguments: [],
+      finalize: true
+    )
+  end
+
+  def down; end
+end
diff --git a/db/schema_migrations/20240130032749 b/db/schema_migrations/20240130032749
new file mode 100644
index 000000000000..3308cdbf3578
--- /dev/null
+++ b/db/schema_migrations/20240130032749
@@ -0,0 +1 @@
+37496ec6cdd7d4f377a5983711b74cc6a58c5dc8bc40f60424bf613e5d848886
\ No newline at end of file

Housekeeper created 2 MRs

How to set up and validate locally

Edited by Dylan Griffith

Merge request reports