Add housekeeper for search migrations (!150553) · Merge requests · GitLab.org / GitLab

Terri Chu requested to merge housekeeper-for-advanced-search-migrations into master Apr 23, 2024

What does this MR do and why?

This MR adds two new housekeeper keeps:

mark advanced search migrations as obsolete
remove obsolete migration code (always leaves the newest obsolete migration)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

N/A

How to set up and validate locally

`DeleteObsoleteAdvancedSearchMigrations`

On master branch there are 2 obsolete migrations. The test run below kept the latest one BackfillHashedRootNamespaceIdOnNotes and removed BackfillHiddenOnIssues

bundle exec gitlab-housekeeper --target-branch=master -k Keeps::DeleteObsoleteAdvancedSearchMigrations -m 1 --dry-run

output

❯ be gitlab-housekeeper --target-branch=master -k Keeps::DeleteObsoleteAdvancedSearchMigrations -m 1 --dry-run
Running keep Keeps::DeleteObsoleteAdvancedSearchMigrations
Merge request URL: (known after create), on branch delete-obsolete-advanced-search-migrations-remove_obsolete.
=> DeleteObsoleteAdvancedSearchMigrations: remove_obsolete
=> Title:
Remove obsolete Advanced search migrations

=> Description:
This migration removes all but the latest obsolete Advanced search migration files from the project.

You can read more about the process for marking Advanced search migrations as obsolete in
https://docs.gitlab.com/ee/development/search/advanced_search_migration_styleguide.html#deleting-advanced-search-migrations-in-a-major-version-upgrade.

As part of our process, we want to ensure all obsolete Advanced search migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html) as obsolete migrations
before removing the migration code from the project. Therefore we can remove code for all Advanced search
migrations that were made obsolete before the last required stop.


=> Attributes:
Labels: maintenance::removal, group::global search, automation:gitlab-housekeeper-authored
Assignees:
Reviewers: sdungarwal

=> Diff:
diff --git a/ee/elastic/migrate/20230209195404_backfill_hidden_on_issues.rb b/ee/elastic/migrate/20230209195404_backfill_hidden_on_issues.rb
deleted file mode 100644
index 8814ff923fde..000000000000
--- a/ee/elastic/migrate/20230209195404_backfill_hidden_on_issues.rb
+++ /dev/null
@@ -1,5 +0,0 @@
-# frozen_string_literal: true
-
-class BackfillHiddenOnIssues < Elastic::Migration
-  include Elastic::MigrationObsolete
-end
diff --git a/ee/elastic/migrate/20230307102400_backfill_hashed_root_namespace_id_on_notes.rb b/ee/elastic/migrate/20230307102400_backfill_hashed_root_namespace_id_on_notes.rb
deleted file mode 100644
index 61d3dd790035..000000000000
--- a/ee/elastic/migrate/20230307102400_backfill_hashed_root_namespace_id_on_notes.rb
+++ /dev/null
@@ -1,20 +0,0 @@
-# frozen_string_literal: true
-
-class BackfillHashedRootNamespaceIdOnNotes < Elastic::Migration
-  include Elastic::MigrationBackfillHelper
-
-  batched!
-  batch_size 9_000
-  throttle_delay 1.minute
-
-  DOCUMENT_TYPE = Note
-  UPDATE_BATCH_SIZE = 100
-
-  private
-
-  def field_name
-    'hashed_root_namespace_id'
-  end
-end
-
-BackfillHashedRootNamespaceIdOnNotes.prepend ::Elastic::MigrationObsolete
diff --git a/ee/spec/elastic/migrate/20230209195404_backfill_hidden_on_issues_spec.rb b/ee/spec/elastic/migrate/20230209195404_backfill_hidden_on_issues_spec.rb
deleted file mode 100644
index d2171884db5f..000000000000
--- a/ee/spec/elastic/migrate/20230209195404_backfill_hidden_on_issues_spec.rb
+++ /dev/null
@@ -1,9 +0,0 @@
-# frozen_string_literal: true
-
-require 'spec_helper'
-require_relative 'migration_shared_examples'
-require File.expand_path('ee/elastic/migrate/20230209195404_backfill_hidden_on_issues.rb')
-
-RSpec.describe BackfillHiddenOnIssues, :elastic_delete_by_query, feature_category: :global_search do
-  it_behaves_like 'a deprecated Advanced Search migration', 20230209195404
-end
diff --git a/ee/spec/elastic/migrate/20230307102400_backfill_hashed_root_namespace_id_on_notes_spec.rb b/ee/spec/elastic/migrate/20230307102400_backfill_hashed_root_namespace_id_on_notes_spec.rb
deleted file mode 100644
index 31bd108e4abd..000000000000
--- a/ee/spec/elastic/migrate/20230307102400_backfill_hashed_root_namespace_id_on_notes_spec.rb
+++ /dev/null
@@ -1,9 +0,0 @@
-# frozen_string_literal: true
-
-require 'spec_helper'
-require_relative 'migration_shared_examples'
-require File.expand_path('ee/elastic/migrate/20230307102400_backfill_hashed_root_namespace_id_on_notes.rb')
-
-RSpec.describe BackfillHashedRootNamespaceIdOnNotes, :elastic_delete_by_query, feature_category: :global_search do
-  it_behaves_like 'a deprecated Advanced Search migration', 20230307102400
-end


Dry run complete. Housekeeper would have created 1 MR on an actual run.

`MarkOldAdvancedSearchMigrationsAsObsolete`

On master branch there are a lot of migrations which are introduced before %16.11 to be marked as obsolete migrations. The test run below marked 3 as obsolete. This has been tested with a run during %17.0 and the keep worked as expected. The only issue seen was the commit message length too long and has been addressed in this MR.

bundle exec gitlab-housekeeper --target-branch=master -k Keeps::MarkOldAdvancedSearchMigrationsAsObsolete -m 3 --dry-run

output

➜ be gitlab-housekeeper --target-branch=master -k Keeps::MarkOldAdvancedSearchMigrationsAsObsolete -m 3 --dry-run
Running keep Keeps::MarkOldAdvancedSearchMigrationsAsObsolete
Merge request URL: (known after create), on branch mark-old-advanced-search-migrations-as-obsolete-mark_obsolete-20230316150000--add-hashed-root-namespace-id-to-merge-requests.
=> MarkOldAdvancedSearchMigrationsAsObsolete: mark_obsolete: 20230316150000: AddHashedRootNamespaceIdToMergeRequests
=> Title:
Mark 20230316150000 as obsolete

=> Description:
This migration marks the 20230316150000 AddHashedRootNamespaceIdToMergeRequests Advanced search migration as obsolete.

This MR will still need changes to remove references to the migration in the code.
At the moment the `gitlab-housekeeper` is not always capable of removing all references so you must check the
diff and pipeline failures to confirm if there are any issues.
It is the responsibility of the assignee (picked from ~"group::global search") to push those changes to this branch.

You can read more about the process for marking Advanced search migrations as obsolete in
https://docs.gitlab.com/ee/development/search/advanced_search_migration_styleguide.html#deleting-advanced-search-migrations-in-a-major-version-upgrade.

As part of our process we want to ensure all Advanced search migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can mark any Advanced search migrations added before the
last required stop as obsolete.


=> Attributes:
Labels: maintenance::refactor, group::global search, automation:gitlab-housekeeper-authored
Assignees: terrichu
Reviewers:

=> Diff:
diff --git a/ee/elastic/docs/20230316150000_add_hashed_root_namespace_id_to_merge_requests.yml b/ee/elastic/docs/20230316150000_add_hashed_root_namespace_id_to_merge_requests.yml
index e93ae747baa0..8e73c828ec9c 100644
--- a/ee/elastic/docs/20230316150000_add_hashed_root_namespace_id_to_merge_requests.yml
+++ b/ee/elastic/docs/20230316150000_add_hashed_root_namespace_id_to_merge_requests.yml
@@ -5,6 +5,6 @@ description: AddHashedRootNamespaceIdToMergeRequests
 group: group::global search
 milestone: '15.10'
 introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/113860
-obsolete: false
+obsolete: true
 marked_obsolete_by_url:
-marked_obsolete_in_milestone:
+marked_obsolete_in_milestone: '17.0'
diff --git a/ee/elastic/migrate/20230316150000_add_hashed_root_namespace_id_to_merge_requests.rb b/ee/elastic/migrate/20230316150000_add_hashed_root_namespace_id_to_merge_requests.rb
index 8a11197321b7..9ee2385fec4a 100644
--- a/ee/elastic/migrate/20230316150000_add_hashed_root_namespace_id_to_merge_requests.rb
+++ b/ee/elastic/migrate/20230316150000_add_hashed_root_namespace_id_to_merge_requests.rb
@@ -15,3 +15,5 @@ def new_mappings
     }
   end
 end
+
+AddHashedRootNamespaceIdToMergeRequests.prepend ::Elastic::MigrationObsolete
diff --git a/ee/spec/elastic/migrate/20230316150000_add_hashed_root_namespace_id_to_merge_requests_spec.rb b/ee/spec/elastic/migrate/20230316150000_add_hashed_root_namespace_id_to_merge_requests_spec.rb
index ed3dffd35df7..ee13956686db 100644
--- a/ee/spec/elastic/migrate/20230316150000_add_hashed_root_namespace_id_to_merge_requests_spec.rb
+++ b/ee/spec/elastic/migrate/20230316150000_add_hashed_root_namespace_id_to_merge_requests_spec.rb
@@ -5,7 +5,5 @@
 require File.expand_path('ee/elastic/migrate/20230316150000_add_hashed_root_namespace_id_to_merge_requests.rb')

 RSpec.describe AddHashedRootNamespaceIdToMergeRequests, :elastic, :sidekiq_inline, feature_category: :global_search do
-  let(:version) { 20230316150000 }
-
-  include_examples 'migration adds mapping'
+  it_behaves_like 'a deprecated Advanced Search migration', 20230316150000
 end


Merge request URL: (known after create), on branch mark-old-advanced-search-migrations-as-obsolete-mark_obsolete-20230317120500--add-hashed-root-namespace-id-to-issues.
=> MarkOldAdvancedSearchMigrationsAsObsolete: mark_obsolete: 20230317120500: AddHashedRootNamespaceIdToIssues
=> Title:
Mark 20230317120500 as obsolete

=> Description:
This migration marks the 20230317120500 AddHashedRootNamespaceIdToIssues Advanced search migration as obsolete.

This MR will still need changes to remove references to the migration in the code.
At the moment the `gitlab-housekeeper` is not always capable of removing all references so you must check the
diff and pipeline failures to confirm if there are any issues.
It is the responsibility of the assignee (picked from ~"group::global search") to push those changes to this branch.

You can read more about the process for marking Advanced search migrations as obsolete in
https://docs.gitlab.com/ee/development/search/advanced_search_migration_styleguide.html#deleting-advanced-search-migrations-in-a-major-version-upgrade.

As part of our process we want to ensure all Advanced search migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can mark any Advanced search migrations added before the
last required stop as obsolete.


=> Attributes:
Labels: maintenance::refactor, group::global search, automation:gitlab-housekeeper-authored
Assignees: sdungarwal
Reviewers:

=> Diff:
diff --git a/ee/elastic/docs/20230317120500_add_hashed_root_namespace_id_to_issues.yml b/ee/elastic/docs/20230317120500_add_hashed_root_namespace_id_to_issues.yml
index 64f7d24f8ce7..b7a9de0c9aa3 100644
--- a/ee/elastic/docs/20230317120500_add_hashed_root_namespace_id_to_issues.yml
+++ b/ee/elastic/docs/20230317120500_add_hashed_root_namespace_id_to_issues.yml
@@ -5,6 +5,6 @@ description: AddHashedRootNamespaceIdToIssues
 group: group::global search
 milestone: '15.11'
 introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/115023
-obsolete: false
+obsolete: true
 marked_obsolete_by_url:
-marked_obsolete_in_milestone:
+marked_obsolete_in_milestone: '17.0'
diff --git a/ee/elastic/migrate/20230317120500_add_hashed_root_namespace_id_to_issues.rb b/ee/elastic/migrate/20230317120500_add_hashed_root_namespace_id_to_issues.rb
index 7033701ecca9..3656d77575ba 100644
--- a/ee/elastic/migrate/20230317120500_add_hashed_root_namespace_id_to_issues.rb
+++ b/ee/elastic/migrate/20230317120500_add_hashed_root_namespace_id_to_issues.rb
@@ -15,3 +15,5 @@ def new_mappings
     }
   end
 end
+
+AddHashedRootNamespaceIdToIssues.prepend ::Elastic::MigrationObsolete
diff --git a/ee/spec/elastic/migrate/20230317120500_add_hashed_root_namespace_id_to_issues_spec.rb b/ee/spec/elastic/migrate/20230317120500_add_hashed_root_namespace_id_to_issues_spec.rb
index 144626854951..d34960cbf562 100644
--- a/ee/spec/elastic/migrate/20230317120500_add_hashed_root_namespace_id_to_issues_spec.rb
+++ b/ee/spec/elastic/migrate/20230317120500_add_hashed_root_namespace_id_to_issues_spec.rb
@@ -5,7 +5,5 @@
 require File.expand_path('ee/elastic/migrate/20230317120500_add_hashed_root_namespace_id_to_issues.rb')

 RSpec.describe AddHashedRootNamespaceIdToIssues, :elastic, :sidekiq_inline, feature_category: :global_search do
-  let(:version) { 20230317120500 }
-
-  include_examples 'migration adds mapping'
+  it_behaves_like 'a deprecated Advanced Search migration', 20230317120500
 end


Merge request URL: (known after create), on branch mark-old-advanced-search-migrations-as-obsolete-mark_obsolete-20230320011400--add-hashed-root-namespace-id-to-commits.
=> MarkOldAdvancedSearchMigrationsAsObsolete: mark_obsolete: 20230320011400: AddHashedRootNamespaceIdToCommits
=> Title:
Mark 20230320011400 as obsolete

=> Description:
This migration marks the 20230320011400 AddHashedRootNamespaceIdToCommits Advanced search migration as obsolete.

This MR will still need changes to remove references to the migration in the code.
At the moment the `gitlab-housekeeper` is not always capable of removing all references so you must check the
diff and pipeline failures to confirm if there are any issues.
It is the responsibility of the assignee (picked from ~"group::global search") to push those changes to this branch.

You can read more about the process for marking Advanced search migrations as obsolete in
https://docs.gitlab.com/ee/development/search/advanced_search_migration_styleguide.html#deleting-advanced-search-migrations-in-a-major-version-upgrade.

As part of our process we want to ensure all Advanced search migrations have had at least one
[required stop](https://docs.gitlab.com/ee/development/database/required_stops.html)
to process the migration. Therefore we can mark any Advanced search migrations added before the
last required stop as obsolete.


=> Attributes:
Labels: maintenance::refactor, group::global search, automation:gitlab-housekeeper-authored
Assignees: maddievn
Reviewers:

=> Diff:
diff --git a/ee/elastic/docs/20230320011400_add_hashed_root_namespace_id_to_commits.yml b/ee/elastic/docs/20230320011400_add_hashed_root_namespace_id_to_commits.yml
index 0f53a6e4a8b6..dcd4c9c0fb4d 100644
--- a/ee/elastic/docs/20230320011400_add_hashed_root_namespace_id_to_commits.yml
+++ b/ee/elastic/docs/20230320011400_add_hashed_root_namespace_id_to_commits.yml
@@ -5,6 +5,6 @@ description: AddHashedRootNamespaceIdToCommits
 group: group::global search
 milestone: '15.11'
 introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/116413
-obsolete: false
+obsolete: true
 marked_obsolete_by_url:
-marked_obsolete_in_milestone:
+marked_obsolete_in_milestone: '17.0'
diff --git a/ee/elastic/migrate/20230320011400_add_hashed_root_namespace_id_to_commits.rb b/ee/elastic/migrate/20230320011400_add_hashed_root_namespace_id_to_commits.rb
index bb2136592d45..d8986cdf1eec 100644
--- a/ee/elastic/migrate/20230320011400_add_hashed_root_namespace_id_to_commits.rb
+++ b/ee/elastic/migrate/20230320011400_add_hashed_root_namespace_id_to_commits.rb
@@ -17,3 +17,5 @@ def new_mappings
     }
   end
 end
+
+AddHashedRootNamespaceIdToCommits.prepend ::Elastic::MigrationObsolete
diff --git a/ee/spec/elastic/migrate/20230320011400_add_hashed_root_namespace_id_to_commits_spec.rb b/ee/spec/elastic/migrate/20230320011400_add_hashed_root_namespace_id_to_commits_spec.rb
index 1f4e8e343197..366ef46cdab5 100644
--- a/ee/spec/elastic/migrate/20230320011400_add_hashed_root_namespace_id_to_commits_spec.rb
+++ b/ee/spec/elastic/migrate/20230320011400_add_hashed_root_namespace_id_to_commits_spec.rb
@@ -5,7 +5,5 @@
 require File.expand_path('ee/elastic/migrate/20230320011400_add_hashed_root_namespace_id_to_commits.rb')

 RSpec.describe AddHashedRootNamespaceIdToCommits, :elastic, :sidekiq_inline, feature_category: :global_search do
-  let(:version) { 20230320011400 }
-
-  include_examples 'migration adds mapping'
+  it_behaves_like 'a deprecated Advanced Search migration', 20230320011400
 end


Dry run complete. Housekeeper would have created 3 MRs on an actual run.

Edited Apr 29, 2024 by Terri Chu

Add housekeeper for search migrations

What does this MR do and why?

MR acceptance checklist

Screenshots or screen recordings

How to set up and validate locally

DeleteObsoleteAdvancedSearchMigrations

MarkOldAdvancedSearchMigrationsAsObsolete

Merge request reports

`DeleteObsoleteAdvancedSearchMigrations`

`MarkOldAdvancedSearchMigrationsAsObsolete`