Skip to content
Snippets Groups Projects

Add migration for backfilling traversal_ids in blobs and wiki blobs

Merged Siddharth Dungarwal requested to merge 351381-backfill-blobs-and-wiki-blobs into master
1 unresolved thread

What does this MR do and why?

Describe in detail what your merge request does and why.

Backfills the traversal_ids for blobs and wiki blobs in the main index, more details in the issue #351381 (closed)

Time for completion estimate calculation (internal link): 278 hours (may take a little longer due to having to work through each project). Indexing will not be paused during the migration.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Run the following query to get the blobs and wiki_blobs with missing traversal_ids
{
  "size": 0,
  "query": {
    "bool": {
      "must_not": {
        "exists": {
          "field": "traversal_ids"
        }
      },
      "must": {
        "terms": {
          "type": [
            "blob",
            "wiki_blob"
          ]
        }
      }
    }
  },
  "aggs": {
    "my-agg-name": {
      "terms": {
        "size": 1000,
        "field": "project_id"
      }
    }
  }
}
  1. Make sure advanced search is enabled and you run the migration from rails console by entering the following lines:
require File.expand_path('ee/elastic/migrate/20221221110300_add_traversal_ids_in_blobs_and_wiki_blobs.rb')
BackfillTraversalIdsToBlobsAndWikiBlobs.new(20221221110300).migrate
  1. Run the query again in ES to verify that there are no records with misssing traversal_ids

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #351381 (closed)

Edited by Terri Chu

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • added 1 commit

    • 588b166e - Add specs and rename migration acc to class name

    Compare with previous version

  • added 1 commit

    • e08e3eb4 - Add specs and rename migration acc to class name

    Compare with previous version

  • Siddharth Dungarwal resolved all threads

    resolved all threads

  • added 1 commit

    • 94d23ae4 - Add update by query and fix specs

    Compare with previous version

  • Siddharth Dungarwal marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

    marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

  • Siddharth Dungarwal resolved all threads

    resolved all threads

  • Siddharth Dungarwal marked this merge request as ready

    marked this merge request as ready

  • requested review from @imand3r

  • added 1 commit

    • d40460c4 - Apply 2 suggestion(s) to 1 file(s)

    Compare with previous version

  • added 2 commits

    • 117263c4 - Add review fixes
    • 50cc88a2 - Merge branch '351381-backfill-blobs-and-wiki-blobs' of...

    Compare with previous version

  • added 1 commit

    • 840f5778 - Merge branch '351381-backfill-blobs-and-wiki-blobs' of...

    Compare with previous version

  • Ian Anderson approved this merge request

    approved this merge request

  • :wave: @imand3r, thanks for approving this merge request.

    This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.

    For more info, please refer to the following links:

  • requested review from @terrichu

  • Terri Chu
  • Terri Chu
  • Terri Chu
  • Terri Chu
  • Terri Chu removed review request for @terrichu

    removed review request for @terrichu

  • I'm updating the type labels to match the issue

  • added 1 commit

    • fac98710 - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • added 2 commits

    • cadb62da - Add migration for backfilling traversal_ids
    • 7241b16b - Merge branch '351381-backfill-blobs-and-wiki-blobs' of...

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • mentioned in merge request !108135 (merged)

  • added 1 commit

    Compare with previous version

  • Siddharth Dungarwal requested review from @terrichu and removed review request for @imand3r

    requested review from @terrichu and removed review request for @imand3r

  • added 1 commit

    • 8d7ad451 - Backfill one project at a time

    Compare with previous version

  • Ian Anderson mentioned in commit 193e2861

    mentioned in commit 193e2861

  • added 1 commit

    • 193e2861 - Add migration for backfilling traversal_ids in a single project

    Compare with previous version

  • Terri Chu
  • Terri Chu removed review request for @terrichu

    removed review request for @terrichu

  • Terri Chu changed the description

    changed the description

  • added 1 commit

    • 1f7f43ff - Apply 4 suggestion(s) to 2 file(s)

    Compare with previous version

  • added 1 commit

    • 0bbede8e - Add spec for multiple projects

    Compare with previous version

  • requested review from @terrichu

  • added 1 commit

    • 051675ce - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • Ian Anderson mentioned in commit d1bce549

    mentioned in commit d1bce549

  • added 1 commit

    • d1bce549 - Add migration for backfilling traversal_ids in a single project

    Compare with previous version

  • mentioned in commit ac497c04

  • added 1 commit

    • ac497c04 - Add migration for backfilling traversal_ids in a single project

    Compare with previous version

    • Resolved by John Mason

      @john-mason Would you mind taking over the backend maintainer reviewer here? I am going to take over as a co-author for any changes required (plus do the monitoring in staging/production/kibana once it merges).

      question: I think this should be ok with the runtime. WDYT?

      thought (non-blocking): elastic_migration_worker is currently disabled in production so I can control when it starts

  • Terri Chu added workflowin review label and removed workflowin dev label

    added workflowin review label and removed workflowin dev label

  • Terri Chu requested review from @john-mason and removed review request for @terrichu

    requested review from @john-mason and removed review request for @terrichu

  • John Mason
  • John Mason
  • John Mason
  • John Mason
  • Terri Chu removed review request for @john-mason

    removed review request for @john-mason

  • Terri Chu added 1 commit

    added 1 commit

    • 1d459168 - Apply maintainer suggestions and refactor specs a bit

    Compare with previous version

  • Terri Chu requested review from @john-mason

    requested review from @john-mason

  • John Mason
  • John Mason
  • John Mason
  • John Mason
  • John Mason approved this merge request

    approved this merge request

  • John Mason resolved all threads

    resolved all threads

  • John Mason enabled an automatic merge when the pipeline for 623ab965 succeeds

    enabled an automatic merge when the pipeline for 623ab965 succeeds

  • merged

  • mentioned in commit 81aa23cb

  • John Mason mentioned in commit 90827cbb

    mentioned in commit 90827cbb

  • added workflowverification label and removed workflowin review label

  • added workflowstaging label and removed workflowcanary label

  • Terri Chu mentioned in merge request !109379 (merged)

    mentioned in merge request !109379 (merged)

  • Chatops says that

    gitlab-org/gitlab!107730 has not been included in the stable branch. The MR will not be released in 15.8.

    So I'll update the milestone to %15.9

  • Dmitry Gruzd changed milestone to %15.9

    changed milestone to %15.9

  • Terri Chu mentioned in merge request !109706 (merged)

    mentioned in merge request !109706 (merged)

  • Terri Chu mentioned in merge request !132173 (merged)

    mentioned in merge request !132173 (merged)

  • Please register or sign in to reply
    Loading