Elasticsearch not indexing new commits to new default branch

Summary

Steps to reproduce

Example Project

Used both GitLab.com and SM Omnibus version 15.6

  1. Create a new project and unselect the "Initialize with README.md" field to have a completely blank project and empty repo.
  2. Clone the empty repository and add a README file. Push the commit to branch main
  3. Search for the file - you should get results
  4. Add a new branch, say new-branch. Change this to be the default branch from main
  5. Make changes to main branch like add some text to README.md file.
  6. Push the commit again to main branch. [Or whichever the non-default branch is now].

image

  1. Switch main branch again to be the default branch. Search for the text you just added in the latest commit. The results do not show up

image

  1. When a new commit is pushed to the main branch afterwards, then search results will appear.

What is the current bug behavior?

IndexStatus does not update when the default branch is switched. Therefore search results after default branch is changed will not appear until a new commit is pushed.

irb(main):001:0> IndexStatus.where('project_id = 126')
=> #<ActiveRecord::Relation [#<IndexStatus id: 480, project_id: 126, indexed_at: "2023-02-02 20:45:33.776672000 +0000", note: nil, last_commit: "602142b8df02eb0adc318445cee9bca67304a675", created_at: "2023-02-02 20:45:33.743883000 +0000", updated_at: "2023-02-02 20:45:33.777140000 +0000", last_wiki_commit: nil, wiki_indexed_at: nil>]>

Note from the screenshot above in step 7, the last_commit should have updated to the HEAD SHA: f5ca7038546bf4406611e0779251c85fdc2e3c2c

elasticsearch.log after updating the default branch to be main again:

{"severity":"DEBUG","time":"2023-02-03T07:08:17.038Z","correlation_id":"01GRB1NFJ21MV7DE6J6KK4P27P","class":"Elastic::ProcessBookkeepingService","redis_set":"elastic:incremental:updates:10:zset","message":"track_items","count":1,"tracked_items_encoded":"[[47997,\"Project 126 project_126\"]]"}
{"severity":"INFO","time":"2023-02-03T07:09:08.883Z","correlation_id":"b3b10879fd2aa03cffc73056b501295b","message":"bulk_indexing_start","redis_set":"elastic:incremental:updates:10:zset","records_count":1,"first_score":47997.0,"last_score":47997.0}
{"severity":"INFO","time":"2023-02-03T07:09:08.920Z","correlation_id":"b3b10879fd2aa03cffc73056b501295b","message":"bulk_submitted","body_size_bytes":600,"bulk_count":1,"errors_count":0}
{"severity":"INFO","time":"2023-02-03T07:09:08.921Z","correlation_id":"b3b10879fd2aa03cffc73056b501295b","message":"bulk_indexer_flushed","flushing_duration_s":0.015760411974042654}
{"severity":"INFO","time":"2023-02-03T07:09:08.921Z","correlation_id":"b3b10879fd2aa03cffc73056b501295b","message":"bulk_indexing_end","redis_set":"elastic:incremental:updates:10:zset","records_count":1,"first_score":47997.0,"last_score":47997.0,"failures_count":0,"bulk_execution_duration_s":0.098619519}
{"severity":"INFO","time":"2023-02-03T07:09:08.921Z","correlation_id":"b3b10879fd2aa03cffc73056b501295b","message":"indexing_done","model_class":"Project","model_id":"126","es_id":"project_126","es_parent":null,"search_indexing_duration_s":300.615688209,"search_indexing_flushing_duration_s":0.015760411974042654}
{"severity":"DEBUG","time":"2023-02-03T07:26:06.476Z","correlation_id":"7f37f64c03a4cb36277a6991676256e1","message":"indexing_commit_range","project_id":110,"from_sha":"666cb9298e1b7c90e3664df28960b13ddb9fe6a2","to_sha":"666cb9298e1b7c90e3664df28960b13ddb9fe6a2","index_wiki":false}
{"severity":"INFO","time":"2023-02-03T07:26:06.656Z","correlation_id":"7f37f64c03a4cb36277a6991676256e1","message":"time=\"2023-02-03T07:26:06Z\" level=info msg=\"Setting timeout\" timeout=24h0m0s\n","status":0,"project_id":110,"from_sha":"666cb9298e1b7c90e3664df28960b13ddb9fe6a2","to_sha":"666cb9298e1b7c90e3664df28960b13ddb9fe6a2","index_wiki":false}
{"severity":"INFO","time":"2023-02-03T07:26:06.665Z","correlation_id":"7f37f64c03a4cb36277a6991676256e1","project_id":110,"wiki":false,"search_indexing_duration_s":0.2135031140060164,"jid":"eb0193caa62a29eb61cd9f

After adding a new commit to the new default branch, the search results appear.

image

And SHA updates:

irb(main):006:0> IndexStatus.where('project_id = 126')
=> #<ActiveRecord::Relation [#<IndexStatus id: 480, project_id: 126, indexed_at: "2023-02-03 08:35:47.619266000 +0000", note: nil, last_commit: "f903252b1020c913b7262125d1fb9f7d4495c6a2", created_at: "2023-02-02 20:45:33.743883000 +0000", updated_at: "2023-02-03 08:35:47.620202000 +0000", last_wiki_commit: nil, wiki_indexed_at: nil>]>

After a new commit elasticsearch.log :

{"severity":"DEBUG","time":"2023-02-03T08:35:47.126Z","correlation_id":"01GRB6NPM5NMABNPM1PJ6Z1BAK","message":"indexing_commit_range","project_id":126,"from_sha":"602142b8df02eb0adc318445cee9bca67304a675","to_sha":"f903252b1020c913b7262125d1fb9f7d4495c6a2","index_wiki":false}
{"severity":"INFO","time":"2023-02-03T08:35:47.509Z","correlation_id":"01GRB6NPM5NMABNPM1PJ6Z1BAK","message":"time=\"2023-02-03T08:35:47Z\" level=info msg=\"Setting timeout\" timeout=24h0m0s\n","status":0,"project_id":126,"from_sha":"602142b8df02eb0adc318445cee9bca67304a675","to_sha":"f903252b1020c913b7262125d1fb9f7d4495c6a2","index_wiki":false}
{"severity":"INFO","time":"2023-02-03T08:35:47.848Z","correlation_id":"01GRB6NPM5NMABNPM1PJ6Z1BAK","project_id":126,"wiki":false,"search_indexing_duration_s":0.7708468890050426,"jid":"4738f5e6244e6e6fedf838cd"}

What is the expected correct behavior?

  1. Indexing updates after the default branch is changed and search results appear in the new main branch.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes