Skip to content

Add migration version to Elasticsearch mapping & config

Terri Chu requested to merge 273234-de-normalize-issue-permission-data-and-add-data-version-to-document into master

What does this MR do?

Related to #273234 (closed)

Based off work in !46672 (merged)

Note: Originally this MR was to add two new fields to the mapping, but one of them already exists (issues_access_level)

This MR does two things:

  • adds an Elasticsearch migration to add a new field to the existing Elasticsearch mapping. The new field will be used to track the migration version for each document in Elasticsearch and allow future migrations to target specific data versions for update. The field MUST be added to the Elasticsearch mapping prior to indexing the data.
  • updates the mapping config so newly created indexes will have it (and will not need to run the migration)

How to test

Note: You need to have Elasticsearch setup, enabled, and have indexing turned on.

I tested this locally by manually kicking off the cron worker in a rails console.

::Elastic::MigrationWorker.new.perform

Monitor the elasticsearch.log file for updates on the migrations.

My Elasticsearch instance has already had the migrations performed before kicking off the worker so they were not run (as expected)

{"severity":"INFO","time":"2020-11-13T13:00:10.387Z","correlation_id":"0ba0fe0268346fc3ccf69f4157356eb3","message":"MigrationWorker: migration[ApplyMaxAnalyzedOffset] did not execute migrate method since it was already executed. Waiting for migration to complete"}
{"severity":"INFO","time":"2020-11-13T13:00:10.396Z","correlation_id":"0ba0fe0268346fc3ccf69f4157356eb3","message":"MigrationWorker: migration[ApplyMaxAnalyzedOffset] updating with completed: true"}
{"severity":"INFO","time":"2020-11-13T13:17:55.504Z","correlation_id":"61af0048580470f23554f55364c5464d","message":"MigrationWorker: migration[AddGitlabMigrationVersionToIndexMapping] did not execute migrate method since it was already executed. Waiting for migration to complete"}
{"severity":"INFO","time":"2020-11-13T13:17:55.517Z","correlation_id":"61af0048580470f23554f55364c5464d","message":"MigrationWorker: migration[AddGitlabMigrationVersionToIndexMapping] updating with completed: true"}

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Terri Chu

Merge request reports