Support all projects being indexed in deletes and transfers

Background

Related to #428070 (closed)

This change only affects instances where Elasticsearch limiting is enabled and configured to limit indexing to specific namespaces or projects. Note: This is how Advanced search is setup on GitLab.com The MR only makes changes to the indexing process but does not touch any code related to search.

This MR is broken into three parts (all behind a disabled feature flag)

Merge request	What it does
!134456 (merged)	Allow all projects being indexed
⭐ !137308 (merged)	Support all projects being indexed when performing deletes and transfers to another namespace
!138098 (merged)	elastic rake task updates

What does this MR do and why?

This MR allows introduces support for project deletions and transfers when all projects are indexed. Indexing all projects is currently behind a disabled feature flag (search_index_all_projects). The main changes are:

• In ElasticDeleteProjectWorker
  • Add option to keep the project in the index. Default behavior is to delete the project record (what happens today).
  • Refactor to delete the project document in a separate call to Elasticsearch.
• Each spot ElasticDeleteProjectWorker is called, use the feature flag to determine if the project document should stay in the index
• spec updates

Screenshots or screen recordings

N/A

How to set up and validate locally

1. Setup elasticsearch in gdk

2. Enable advanced search and Elasticsearch indexing restrictions, I chose one group and project

   

3. Disable the feature flag in rails console: Feature.disable(:search_index_all_projects)

4. Reindex everything from scratch: bundle exec rake gitlab:elastic:index

5. Verify that only the expected projects exist in the index, in my case that is 3 projects (2 under the selected group, and 1 selected project)

      curl --request GET \
     --url http://localhost:9200/gitlab-development-projects/_search \
     --data '{
   	"query": {
   		"bool": {
   			"must": [
   				{
   					"term": {
   						"type": {
   							"value": "project"
   						}
   					}		
   				}
   	    	]
   		}
   	}
   }'

6. create a two new projects, add at least 1 issue (or merge request or comment) in each

   • indexed_project = one in an indexed namespace (from the ones you picked above)
   • not_indexed_project = one in a non-indexed namespace

7. enable the feature flag for the not_indexed_project project: Feature.enable(:search_index_all_projects, not_indexed_project.root_namespace)

8. update the not_indexed_project project settings (I updated the description)

9. verify in the log/elasticsearch.log that ONLY the project record is queued for indexing (not any associated records)

10. move not_indexed_project to an indexed namespace

11. verify in the log/elasticsearch.log that the project and all associated data is queued for indexing

12. search and verify that the issue records exist in the index (Elasticsearch queries to verify data are included below)

13. move the project back to a non indexed namespace

14. verify in the log/elasticsearch.log that ONLY the project record is queued for indexing (not any associated records)

15. search and verify that the issue records do not exist in the index

16. delete the project fully: ProjectDestroyWorker.new.perform(35, User.first.id, {})

17. search and verify that the records do not exist in the index

Elasticsearch queries to verify data

Projects

curl --request GET \
  --url http://localhost:9200/gitlab-development-projects/_search \
  --header 'Content-Type: application/json' \
  --data '{
	"query": {
		"bool": {
			"must": [
				{
					"term": {
						"id": {
							"value": 19
						}
					}
				}
			]
		}
	}
}'

Issues

```
curl --request POST \
  --url http://localhost:9200/gitlab-development-issues/_search \
  --header 'Content-Type: application/json' \
  --data '{
	"query": {
		"bool": {
			"must": [
				{
					"term": {
						"project_id": 19
					}
				}
			]
		}
	}
}'
```

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.