Add users to Elasticsearch
Context
In order to use Elasticsearch to perform user search for better performance and experience as part of &8980, the first step is to create an Elasticsearch index for users.
This MR
- Adds a migration to create a user index in elasticsearch
- Adds three elastic lib files:
UserInstanceProxy
,UserConfig
andUserClassProxy
(empty but required, will be implemented in #377587 (closed))- Notable field is
namespace_ancestry_ids
: it takes the namespace_ids of all projects the user is a member of and appendsp#{project.id}-
and the namespace_ids of all groups the user is a member of. This enables us to filter by projects and groups.
- Notable field is
- Prepends
Elastic::ApplicationVersionedSearch
onee/user
model to keep documents in elasticsearch up to date - Creates a feature flag to control when users are added to elasticsearch
- Specs
Next steps
- Verify migration runs successfully on staging and production and that index has been created
- Do a partial rollout of the
index_user_callback
feature flag on gitlab.com #378364 (closed) - Create a migration to backfill users
- Implement the searching part guarded by feature flag and
migration_has_finished?
#377587 (closed) - Enable the indexing feature flag by default
How to set up and validate locally
Creating the index using the migration
- Enable Advanced Search
- Trigger
Elastic::MigrationWorker.new.perform
repeatedly - Verify that the new migration has been run via
curl http://localhost:9200/gitlab-development-migrations/_doc/20221018125700
. Completed should betrue
. - Verify that you have a
gitlab-development-users-<timestamp>
index viacurl http://localhost:9200/_cat/indices
- Verify that you have a
gitlab-development-users
alias viacurl http://localhost:9200/_cat/aliases
- Check that no documents exist in the user index (because the feature flag is disabled):
curl http://localhost:9200/gitlab-development-users/_search
should return no hits. - Verify mappings via
curl http://localhost:9200/gitlab-development-users/_mappings
- Verify settings via
curl http://localhost:9200/gitlab-development-users/_settings
Verify the feature flag default state
- Update a user record. For example: in a
rails c
runUser.first.update(name: "Test User")
- Run
ElasticIndexInitialBulkCronWorker.new.perform
andElasticIndexBulkCronWorker.new.perform
on repeat - Check that no documents exist in the user index after the update (because the feature flag is still disabled):
curl http://localhost:9200/gitlab-development-users/_search
should return no hits.
Verify enabling feature flag per user
- Enable the feature flag for the user: in a
rails c
runFeature.enable(:index_user_callback, User.first)
- Update a user record. For example: in a
rails c
runUser.first.update(name: "Test User")
- Run
ElasticIndexInitialBulkCronWorker.new.perform
andElasticIndexBulkCronWorker.new.perform
on repeat - Check that the document has been created in the user index after the update (because the feature flag is enabled for this user):
curl http://localhost:9200/gitlab-development-users/_search
should return one hit.
Verify enabling feature flag for all actors
- Enable the feature flag for all actors: in a
rails c
runFeature.enable(:index_user_callback)
- Update a user record. For example: in a
rails c
runUser.last.update(name: "Test User")
- Run
ElasticIndexInitialBulkCronWorker.new.perform
andElasticIndexBulkCronWorker.new.perform
on repeat - Check that a document has been created in the user index after the update:
curl http://localhost:9200/gitlab-development-users/_search
should return more than one hit.
Outcome
An elasticsearch index for users with fields that enable us to search on.
Example documents:
Outcome of successful migration:
Outcome of unsuccessful migration (when max attempts is exceeded):
Edited by Madelein van Niekerk