Add users to Elasticsearch

Context

In order to use Elasticsearch to perform user search for better performance and experience as part of &8980, the first step is to create an Elasticsearch index for users.

This MR

• Adds a migration to create a user index in elasticsearch
• Adds three elastic lib files: UserInstanceProxy, UserConfig and UserClassProxy (empty but required, will be implemented in #377587 (closed))
  • Notable field is namespace_ancestry_ids: it takes the namespace_ids of all projects the user is a member of and appends p#{project.id}- and the namespace_ids of all groups the user is a member of. This enables us to filter by projects and groups.
• Prepends Elastic::ApplicationVersionedSearch on ee/user model to keep documents in elasticsearch up to date
• Creates a feature flag to control when users are added to elasticsearch
• Specs

Next steps

• Verify migration runs successfully on staging and production and that index has been created
• Do a partial rollout of the index_user_callback feature flag on gitlab.com #378364 (closed)
• Create a migration to backfill users
• Implement the searching part guarded by feature flag and migration_has_finished? #377587 (closed)
• Enable the indexing feature flag by default

How to set up and validate locally

Creating the index using the migration

1. Enable Advanced Search
2. Trigger Elastic::MigrationWorker.new.perform repeatedly
3. Verify that the new migration has been run via curl http://localhost:9200/gitlab-development-migrations/_doc/20221018125700. Completed should be true.
4. Verify that you have a gitlab-development-users-<timestamp> index via curl http://localhost:9200/_cat/indices
5. Verify that you have a gitlab-development-users alias via curl http://localhost:9200/_cat/aliases
6. Check that no documents exist in the user index (because the feature flag is disabled): curl http://localhost:9200/gitlab-development-users/_search should return no hits.
7. Verify mappings via curl http://localhost:9200/gitlab-development-users/_mappings
8. Verify settings via curl http://localhost:9200/gitlab-development-users/_settings

Verify the feature flag default state

1. Update a user record. For example: in a rails c run User.first.update(name: "Test User")
2. Run ElasticIndexInitialBulkCronWorker.new.perform and ElasticIndexBulkCronWorker.new.perform on repeat
3. Check that no documents exist in the user index after the update (because the feature flag is still disabled): curl http://localhost:9200/gitlab-development-users/_search should return no hits.

Verify enabling feature flag per user

1. Enable the feature flag for the user: in a rails c run Feature.enable(:index_user_callback, User.first)
2. Update a user record. For example: in a rails c run User.first.update(name: "Test User")
3. Run ElasticIndexInitialBulkCronWorker.new.perform and ElasticIndexBulkCronWorker.new.perform on repeat
4. Check that the document has been created in the user index after the update (because the feature flag is enabled for this user): curl http://localhost:9200/gitlab-development-users/_search should return one hit.

Verify enabling feature flag for all actors

1. Enable the feature flag for all actors: in a rails c run Feature.enable(:index_user_callback)
2. Update a user record. For example: in a rails c run User.last.update(name: "Test User")
3. Run ElasticIndexInitialBulkCronWorker.new.perform and ElasticIndexBulkCronWorker.new.perform on repeat
4. Check that a document has been created in the user index after the update: curl http://localhost:9200/gitlab-development-users/_search should return more than one hit.

Outcome

An elasticsearch index for users with fields that enable us to search on.

Example documents:

Outcome of successful migration:

Outcome of unsuccessful migration (when max attempts is exceeded):