Skip to content

Queue projects for indexing when missing from the index

Terri Chu requested to merge 214601-repair-projects-missing-from-index into master

What does this MR do and why?

Related to #214601 (closed)

For projects where the documents are missing from the project index, queue the record for indexing. This will only queue the project record (not any associated records or repository data)

Screenshots or screen recordings

N/A

How to set up and validate locally

  1. setup gdk for elasticsearch and index everything

  2. manually delete a project from the index

    curl --request POST --url http://localhost:9200/gitlab-development/_delete_by_query --header 'Content-Type: application/json' --data '{ "query": { "bool": { "must": [ { "term": { "type": { "value": "project" } } }, { "term": { "id": { "value": 1 } } } ] } } }'

  3. verify project is not in the index (should get a 404)

    curl --request GET \
  --url http://localhost:9200/gitlab-development/_doc/project_1

  4. open rails console, run the index repair service, and the bulk indexing cronWorker

    project = Project.find(1)
Search::IndexRepairService.execute(project) # queue up project for indexing
Elastic::ProcessBookkeepingService.new.execute # force indexing to empty queue

  5. verify project is in the index (should get a document back)

    curl --request GET \
  --url http://localhost:9200/gitlab-development/_doc/project_1

  6. logs from log/elasticsearch.log should show the warning log about project document missing from index

    {"severity":"WARN","time":"2023-05-17T23:51:29.860Z","class":"Search::IndexRepairService","message":"project document missing from index","namespace_id":22,"root_namespace_id":22,"project_id":1}
{"severity":"DEBUG","time":"2023-05-17T23:51:29.860Z","class":"Elastic::ProcessBookkeepingService","redis_set":"elastic:incremental:updates:1:zset","message":"track_items","count":1,"tracked_items_encoded":"[[25,\"Project 1 project_1\"]]"}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports