Add --filter-identifiers option to gitlab-housekeeper
What does this MR do and why?
During local usage of this we found it is often desirable to get the housekeeper to generate a specific MR and ignore the other ones. Since it just does them in order we had to previously manually hack the keeps to skip the changes we don't care about.
This change introduces a command line option --filter-identifiers
which takes a comma separated list which will be matched against the
list of "identifiers" in a change yielded from a keep. The identifiers
is an array so to allow for flexible regex matching of specific elements
in the array we filter such that all regexes must match at least one of
the identifiers in the list.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Output from running with a filter
$ bundle exec gitlab-housekeeper -m2 -d -r keeps/overdue_finalize_background_migration.rb -k Keeps::OverdueFinalizeBackgroundMigration --filter-identifiers 'Backfill.*Reads'
create db/post_migrate/20240130032730_finalize_backfill_finding_id_in_vulnerabilities.rb
Skipping change: ["OverdueFinalizeBackgroundMigration", "BackfillFindingIdInVulnerabilities"] due to not matching filter
create db/post_migrate/20240130032740_finalize_backfill_has_merge_request_of_vulnerability_reads.rb
=> OverdueFinalizeBackgroundMigration: BackfillHasMergeRequestOfVulnerabilityReads
=> Title:
Finalize migration BackfillHasMergeRequestOfVulnerabilityReads
=> Description:
This migration was finished at `2023-09-19 05:04:03 UTC`, you can confirm
the status using our
[batched background migration chatops commands](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#monitor-the-progress-and-status-of-a-batched-background-migration).
To confirm it is finished you can run:
/chatops run batched_background_migrations status 1000456
The last time this background migration was triggered was in [db/post_migrate/20230907155247_queue_backfill_has_merge_request_of_vulnerability_reads.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/post_migrate/20230907155247_queue_backfill_has_merge_request_of_vulnerability_reads.rb)
You can read more about the process for finalizing batched background migrations in
https://docs.gitlab.com/ee/development/database/batched_background_migrations.html .
As part of our process we want to ensure all batched background migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can finalize any batched background migration that was added before the
last required stop.
=> Attributes:
Labels: automation:gitlab-housekeeper-authored
=> Diff:
diff --git a/db/docs/batched_background_migrations/backfill_has_merge_request_of_vulnerability_reads.yml b/db/docs/batched_background_migrations/backfill_has_merge_request_of_vulnerability_reads.yml
index d6cd709ca70a..a2190506fca0 100644
--- a/db/docs/batched_background_migrations/backfill_has_merge_request_of_vulnerability_reads.yml
+++ b/db/docs/batched_background_migrations/backfill_has_merge_request_of_vulnerability_reads.yml
@@ -4,3 +4,4 @@ description: Backfills has_merge_request column for vulnerability_reads table.
feature_category: database
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/130952
milestone: '16.4'
+finalized_by: '20240130032740'
diff --git a/db/post_migrate/20240130032740_finalize_backfill_has_merge_request_of_vulnerability_reads.rb b/db/post_migrate/20240130032740_finalize_backfill_has_merge_request_of_vulnerability_reads.rb
new file mode 100644
index 000000000000..c76125bd14d2
--- /dev/null
+++ b/db/post_migrate/20240130032740_finalize_backfill_has_merge_request_of_vulnerability_reads.rb
@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+class FinalizeBackfillHasMergeRequestOfVulnerabilityReads < Gitlab::Database::Migration[2.2]
+ milestone '16.9'
+
+ disable_ddl_transaction!
+
+ restrict_gitlab_migration gitlab_schema: :gitlab_main
+
+ def up
+ ensure_batched_background_migration_is_finished(
+ job_class_name: 'BackfillHasMergeRequestOfVulnerabilityReads',
+ table_name: :vulnerability_reads,
+ column_name: :vulnerability_id,
+ job_arguments: [],
+ finalize: true
+ )
+ end
+
+ def down; end
+end
diff --git a/db/schema_migrations/20240130032740 b/db/schema_migrations/20240130032740
new file mode 100644
index 000000000000..8dc2e01adab1
--- /dev/null
+++ b/db/schema_migrations/20240130032740
@@ -0,0 +1 @@
+708890d332b90cb5be8b6232fe7c1f9ad0ede42682ca7f2a567eb939bde61682
\ No newline at end of file
create db/post_migrate/20240130032749_finalize_backfill_has_remediations_of_vulnerability_reads.rb
=> OverdueFinalizeBackgroundMigration: BackfillHasRemediationsOfVulnerabilityReads
=> Title:
Finalize migration BackfillHasRemediationsOfVulnerabilityReads
=> Description:
This migration was finished at `2024-01-25 11:23:05 UTC`, you can confirm
the status using our
[batched background migration chatops commands](https://docs.gitlab.com/ee/development/database/batched_background_migrations.html#monitor-the-progress-and-status-of-a-batched-background-migration).
To confirm it is finished you can run:
/chatops run batched_background_migrations status 1000469
The last time this background migration was triggered was in [db/post_migrate/20231201204712_requeue2_backfill_has_remediations_of_vulnerability_reads.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/post_migrate/20231201204712_requeue2_backfill_has_remediations_of_vulnerability_reads.rb)
You can read more about the process for finalizing batched background migrations in
https://docs.gitlab.com/ee/development/database/batched_background_migrations.html .
As part of our process we want to ensure all batched background migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can finalize any batched background migration that was added before the
last required stop.
=> Attributes:
Labels: automation:gitlab-housekeeper-authored
=> Diff:
diff --git a/db/docs/batched_background_migrations/backfill_has_remediations_of_vulnerability_reads.yml b/db/docs/batched_background_migrations/backfill_has_remediations_of_vulnerability_reads.yml
index 18dd61379214..327b913ce837 100644
--- a/db/docs/batched_background_migrations/backfill_has_remediations_of_vulnerability_reads.yml
+++ b/db/docs/batched_background_migrations/backfill_has_remediations_of_vulnerability_reads.yml
@@ -1,9 +1,10 @@
---
migration_job_name: BackfillHasRemediationsOfVulnerabilityReads
-description: Backfills has_remediations column for vulnerability_reads table.
- Originally introduced via https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133714
- RE-ran because there was a error in remediation ingestion logic.
+description: Backfills has_remediations column for vulnerability_reads table. Originally
+ introduced via https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133714 RE-ran
+ because there was a error in remediation ingestion logic.
feature_category: database
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/133714
milestone: '16.7'
queued_migration_version: 20231201204712
+finalized_by: '20240130032749'
diff --git a/db/post_migrate/20240130032749_finalize_backfill_has_remediations_of_vulnerability_reads.rb b/db/post_migrate/20240130032749_finalize_backfill_has_remediations_of_vulnerability_reads.rb
new file mode 100644
index 000000000000..25dad5ecabe1
--- /dev/null
+++ b/db/post_migrate/20240130032749_finalize_backfill_has_remediations_of_vulnerability_reads.rb
@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+class FinalizeBackfillHasRemediationsOfVulnerabilityReads < Gitlab::Database::Migration[2.2]
+ milestone '16.9'
+
+ disable_ddl_transaction!
+
+ restrict_gitlab_migration gitlab_schema: :gitlab_main
+
+ def up
+ ensure_batched_background_migration_is_finished(
+ job_class_name: 'BackfillHasRemediationsOfVulnerabilityReads',
+ table_name: :vulnerability_reads,
+ column_name: :vulnerability_id,
+ job_arguments: [],
+ finalize: true
+ )
+ end
+
+ def down; end
+end
diff --git a/db/schema_migrations/20240130032749 b/db/schema_migrations/20240130032749
new file mode 100644
index 000000000000..3308cdbf3578
--- /dev/null
+++ b/db/schema_migrations/20240130032749
@@ -0,0 +1 @@
+37496ec6cdd7d4f377a5983711b74cc6a58c5dc8bc40f60424bf613e5d848886
\ No newline at end of file
Housekeeper created 2 MRs