Projects become unindexed
Summary
Large deployments sometime don't automatically index all non-empty projects into Opensearch. This seems to be a fairly regular occurrence on deployments with large numbers of projects that are growing steadily (20K+). On two customers this has started recurring regularly
Steps to reproduce
Let the customer add projects and wait.
What is the current bug behavior?
A customer creates some projects, most get indexed. Eventually except 1 or 2 projects start becoming non-indexed
What is the expected correct behavior?
All projects with content should be indexed for global search
Relevant logs and/or screenshots
Identifying missing non-empty projects:
(sum(elasticsearch_indices_docs_primary{job="prometheus-elasticsearch-exporter",service=~"advanced-search-elasticsearch-exporter.+"} * on(index) group_left(alias) elasticsearch_indices_aliases{alias="gitlab-production-projects"}) - sum(gitlab_database_rows{query_name="projects"})) < 0
Using rake gitlab-rake gitlab:elastic:projects_not_indexed
will show empty projects that shouldn't be indexed.
Identifying the missing project(s):
not_indexed_and_not_empty = []
::Search::ElasticProjectsNotIndexedFinder.execute.each_batch do |batch|
batch.inc_routes.each do |project|
if !project.empty_repo?
not_indexed_and_not_empty << project
end
end
end
if not_indexed_and_not_empty.present?
not_indexed_and_not_empty.each { |p| puts "Project '#{p.full_path}' (ID: #{p.id}) is not indexed and not empty.".color(:red) }
end
In the two large customers I see this occurring, I have 60 days of data to analyze. In 1 customer I see 17 times over 60 days where at least 1 project wasn't indexed within 8 hours:
Possible fixes
cc @johnmason