Reindex GitLab.com Global Search Elasticsearch cluster
C3
Production Change - Criticality 3Change Objective | Describe the objective of the change |
---|---|
Change Type | ConfigurationChange |
Services Impacted | GitLab.com , Redis SharedState, Redis Sidekiq, Elasticsearch cluster, Sidekiq |
Change Team Members | @DylanGriffith |
Change Criticality | C3 |
Change Reviewer or tested in staging | |
Dry-run output | |
Due Date | 2020-06-04 00:32 UTC |
Time tracking |
Detailed steps for the change
Pre-check
-
Run all the steps on staging -
Confirm the cluster storage is less than 33% full (more than 67% free)
Process
-
Confirm the cluster storage is less than 33% full (more than 67% free) -
Let SRE on call know that we are triggering the re-index in #production: @sre-oncall please note we are doing a reindex of our production Elasticsearch cluster which will re-index all of our production global search index to another index in the same cluster using the Elasticsearch reindex API. During the reindex we'll be pausing indexing to the cluster which will cause the incremental updates queue to grow but should not cause alerts since we don't have any for that queue. This will increase load on the Elasticsearch cluster but should not impact any other systems. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2213
-
Pause indexing writes: Admin > Settings > Integrations > Elasticsearch > Pause Elasticsearch indexing
-
Take a snapshot of the cluster -
In any console set CLUSTER_URL
and confirm that it is the expected cluster with expected indices:-
curl $CLUSTER_URL/_cat/indices
-
-
Note the total size of the source gitlab-production-202004062333
index:4.9 TB
-
Note the total number of documents in the source gitlab-production-202004062333
index:127170026
curl $CLUSTER_URL/gitlab-production-202004062333/_count
-
Create new destination index gitlab-production-202006040000
with correct settings/mappings from rails console:-
Gitlab::Elastic::Helper.new(target_name: 'gitlab-production-202006040000').create_empty_index(with_alias: false)
-
-
Set index settings in destination index to optimize for writes: curl -XPUT -d '{"index":{"number_of_replicas":"0","refresh_interval":"-1","translog":{"durability":"async"}}}' -H 'Content-Type: application/json' "$CLUSTER_URL/gitlab-production-202006040000/_settings"
-
Trigger re-index from source index gitlab-production-202004062333
to destination indexgitlab-production-202006040000
-
curl -H 'Content-Type: application/json' -d '{ "source": { "index": "gitlab-production-202004062333" }, "dest": { "index": "gitlab-production-202006040000" }, "script": { "source": "ctx._source.remove(\"file_name\"); ctx._source.remove(\"content\");" } }' -X POST "$CLUSTER_URL/_reindex?slices=auto&wait_for_completion=false"
-
-
Note the returned task ID from the above: kYs7t1EZTcGOQ_FTrLVubQ:1485991
-
Note the time when the task started: 2020-06-04 01:09:22 UTC
-
Track the progress of reindexing using the Tasks API curl $CLUSTER_URL/_tasks/$TASK_ID
-
Note the time when the task finishes: 2020-06-04 08:33:10 UTC
-
Note the total time taken to reindex: 7.4 hrs
-
Change the refresh_interval
setting on Destination Cluster to60
-
curl -XPUT -d '{"index":{"refresh_interval":"60s"}}' -H 'Content-Type: application/json' "$CLUSTER_URL/gitlab-production-202006040000/_settings"
-
-
Verify number of documents in Destination index
=number of documents in Source index
-
Be aware it may take 60s to refresh on the destination cluster -
curl $CLUSTER_URL/gitlab-production-202004062333/_count
=>127170026
-
curl $CLUSTER_URL/gitlab-production-202006040000/_count
=>127170026
-
-
Force merge the index to speed up replication curl -XPOST $CLUSTER_URL/gitlab-production-202006040000/_forcemerge
-
Increase replication on Destination index to 1
:-
curl -XPUT -d '{"index":{"number_of_replicas":"1"}}' -H 'Content-Type: application/json' "$CLUSTER_URL/gitlab-production-202006040000/_settings"
-
-
Increase recovery max bytes to speed up replication: -
curl -H 'Content-Type: application/json' -d '{"persistent":{"indices.recovery.max_bytes_per_sec": "200mb"}}' -XPUT $CLUSTER_URL/_cluster/settings
-
-
Wait for cluster monitoring to show the replication has completed -
Set recovery max bytes back to default -
curl -H 'Content-Type: application/json' -d '{"persistent":{"indices.recovery.max_bytes_per_sec": null}}' -XPUT $CLUSTER_URL/_cluster/settings
-
-
Set translog
durability back torequest
on the destination index back to default:curl -XPUT -d '{"index":{"translog":{"durability":"request"}}}' -H 'Content-Type: application/json' "$CLUSTER_URL/gitlab-production-202006040000/_settings"
-
Note the size of the destination index gitlab-production-202004062333
index:1.1 TB
-
Update the alias gitlab-production
to point to the new index-
curl -XPOST -H 'Content-Type: application/json' -d '{"actions":[{"add":{"index":"gitlab-production-202006040000","alias":"gitlab-production"}}, {"remove":{"index":"gitlab-production-202004062333","alias":"gitlab-production"}}]}' $CLUSTER_URL/_aliases
-
Confirm it works curl $CLUSTER_URL/gitlab-production/_count
-
-
Test that prefix searching now behaves as expected per gitlab-org/gitlab#27918 (comment 324005806) -
Unpause indexing writes: Admin > Settings > Integrations > Elasticsearch > Pause Elasticsearch indexing
-
Wait until the backlog of incremental updates gets below 10,000 - Chart
Global search incremental indexing queue depth
https://dashboards.gitlab.net/d/sidekiq-main/sidekiq-overview?orgId=1
- Chart
-
Create a comment somewhere then search for it to ensure indexing still works (can take up to 2 minutes before it shows up in the search results) -
Delete the old gitlab-production-202004062333
index-
curl -XDELETE $CLUSTER_URL/gitlab-production-202004062333
-
-
Test again that searches work as expected -
Scale the cluster down again based on the current size
Monitoring
Key metrics to observe
- Gitlab admin panel:
- Grafana:
- Platform triage
- Sidekiq:
-
Sidekiq SLO dashboard overview
-
Sidekiq Queue Lengths per Queue
- Expected to climb during initial indexing and sudden drop-off once initial indexing jobs are finished.
Sidekiq Inflight Operations by Queue
-
Node Maximum Single Core Utilization per Priority
- expected to be 100% during initial indexing
-
-
Sidekiq SLO dashboard overview
- Redis-sidekiq:
-
Redis-sidekiq SLO dashboard overview
-
Memory Saturation
- If memory usage starts growing rapidly it might get OOM killed (which would be really bad because we would lose all other queued jobs, for example scheduled CI jobs).
- If it gets close to SLO levels, the rate of growth should be evaluated and the indexing should potentially be stopped.
-
-
Redis-sidekiq SLO dashboard overview
- Incremental updates queue:
- Chart
Global search incremental indexing queue depth
https://dashboards.gitlab.net/d/sidekiq-main/sidekiq-overview?orgId=1 - From rails console
Elastic::ProcessBookkeepingService.queue_size
- Chart
Other metrics to observe
- Elastic support diagnostics: In the event of any issues (eg. like last time seeing too many threads in https://support.elastic.co/customers/s/case/5004M00000cL8IJ/cluster-crashing-under-high-load-possibly-due-to-jvm-heap-size-too-large-again) we can grab the support diagnostics per instructions at https://support.elastic.co/customers/s/article/support-diagnostics
- Grafana:
- Rails:
- Postgres:
- patroni SLO dashboard overview
- postgresql overview
- pgbouncer SLO dashboard overview
- pgbouncer overview
-
"Waiting Sidekiq pgbouncer Connections"
- If we see this increase to say 500 and stay that way then we should be concerned and disable indexing at that point
- Gitaly:
- Gitaly SLO dashboard overview
- Gitaly latency
- Gitaly saturation overview
-
Gitaly single node saturation
- If any nodes on this graph are maxed out for a long period of time correlated with enabling this we should disable it. We should first confirm by shutting down
ElasticCommitIndexerWorker
that it will help and then stop if it's clearly correlated.
- If any nodes on this graph are maxed out for a long period of time correlated with enabling this we should disable it. We should first confirm by shutting down
Rollback steps
-
If you get past the step of updating the alias then simply switch the Alias to point back to the original index -
curl -XPOST -H 'Content-Type: application/json' -d '"actions":[{"add":{"index":"gitlab-production-202004062333","alias":"gitlab-production"}}, {"remove":{"index":"gitlab-production-202006040000,"alias":"gitlab-production"}}]}' $CLUSTER_URL/_aliases
-
Confirm it works curl $CLUSTER_URL/gitlab-production/_count
-
-
Ensure any updates that only went to Destination index are replayed against Source Cluster by searching the logs for the updates https://gitlab.com/gitlab-org/gitlab/-/blob/e8e2c02a6dbd486fa4214cb8183d428102dc1156/ee/app/services/elastic/process_bookkeeping_service.rb#L23 and triggering those updates again using ProcessBookkeepingService#track
Changes checklist
-
Detailed steps and rollback steps have been filled prior to commencing work -
Person on-call has been informed prior to change being rolled out
Edited by Dylan Griffith