Skip to content

Add schema_version to main index mapping

Terri Chu requested to merge 214601-add-schema_version-to-main-index into master

What does this MR do and why?

Related to #214601 (closed)

Adding schema_version to index mapping for the main index (contains, project, wiki_blobs, and blob document types).

Note: We would like to have a schema_version field available for future migrations in all document types.

This is prep work to be able to easily backfill traversal_ids for projects which will be needed for the IndexIntegrityWorker. The IndexIntegrityWorker (dev in progress) will need to pull all projects for a root namespace and I need the traversal_ids field populated for that. Currently it's only being populated for blobs and wiki_blobs.

Screenshots or screen recordings

N/A

How to set up and validate locally

prerequisites

setup Elasticsearch in gdk

check config.rb changes

  1. create an index from scratch: bundle exec rake gitlab:elastic:index
  2. verify the new mapping is there and has data (tests the config.rb changes) GET http://localhost:9200/gitlab-development/_search
{
	"query": {
		"bool": {
			"must": [
				{
					"term": {
						"type": {
							"value": "project"
						}
					}
				}
			]
		}
	}
}

migration changes

  1. checkout the master branch
  2. recreate the indexes from scratch : bundle exec rake gitlab:elastic:index
  3. add a new project
  4. verify that it doesn't throw an error and indexes properly (without the new field)
  5. stop rails-background-jobs to prevent it from running the migration automatically: gdk stop rails-background-jobs
  6. start rails console
  7. run the migration in the rails console
  8. verify that the schema_version field exists in the mappings: GET http://localhost:9200/gitlab-development/_mappings
  9. add a new or update an existing project and run the indexing manually: Elastic::ProcessBookkeepingService.new.execute
  10. verify that the schema_version field exists and is populated for the project in the search response GET http://localhost:9200/gitlab-development/_search
{
	"query": {
		"bool": {
			"must": [
				{
					"term": {
						"type": {
							"value": "project"
						}
					}
				}
			]
		}
	}
}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Terri Chu

Merge request reports