Reindex GitLab.com Global Search Elasticsearch cluster main index using Zero-Downtime reindexing
Production Change
Change Summary
We have a list of changes we want to apply to GitLab.com main Advanced Search index:
- gitlab-org/gitlab#349099 (closed) (gitlab-org/gitlab!77226 (merged))
- gitlab-org/gitlab#346914 (closed) (gitlab-org/gitlab!96785 (merged))
- gitlab-org/gitlab#371988 (closed)
This can be done by reindexing the index. Only execute this change request after gitlab-org/gitlab!100424 (merged) has reached production
Change Details
- Services Impacted - Elasticsearch global search
- Change Technician - @dgruzd (EMEA) @john-mason (AMER)
- Change Reviewer - @terrichu
- Time tracking - 48h
- Downtime Component - No downtime, but Advanced Search indexing will be paused
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 60
-
Run all the steps on staging
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 60 to trigger reindexing
-
Set label changein-progress /label ~change::in-progress
-
Add a silence via https://alerts.gitlab.net/#/silences/new with a matcher on alert name: env="gprd"
,alertname="SearchServiceElasticsearchIndexingTrafficAbsent"
,alertname="gitlab_search_indexing_queue_backing_up"
, andalertname="SidekiqServiceGlobalSearchIndexingApdexSLOViolation"
. Link the comment field back to the Change Request Issue. - https://alerts.gitlab.net/#/silences/782b1865-69e4-482d-b76d-b3ca418a7a51 -
Let SRE on call know that we are triggering the re-index in #production: @sre-oncall please note we are doing a reindex of one of our production Elasticsearch cluster indices which will re-index all of our main index in production global search index to another index in the same cluster using the Elasticsearch reindex API. During the reindex we'll be pausing indexing to the cluster which will cause the incremental updates queue to grow. We have added a silence for the SearchServiceElasticsearchIndexingTrafficAbsent alert. This will increase load on the Elasticsearch cluster but should not impact any other systems. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7879
-
Take a screenshot of index advanced metrics for last 30 days and last 7 days and attach to a comment on this issue -
Update the number of shards for the main index ( Elastic::IndexSetting.find_by(alias_name: 'gitlab-production').update!(number_of_shards: 200)
) -
Scale up the cluster up by 5.81 TB x 2 -
Wait until shards are finished reallocating -
Trigger re-index Elastic::ReindexingTask.create!(targets: %w[Repository])
-
Monitor the status of the reindexing through rails console Elastic::ReindexingTask.current
-
Ensure that it has finished successfully -
Note the time when the task finishes: TODO -
Note the total time taken to reindex: TODO -
Wait until the backlog of incremental updates gets below 10,000 - Chart
Global search incremental indexing queue depth
https://dashboards.gitlab.net/d/sidekiq-main/sidekiq-overview?orgId=1
- Chart
-
Create a file somewhere then search for it to ensure indexing still works (can take up to 2 minutes before it shows up in the search results) -
remove the alert silences -
Set label changecomplete /label ~change::complete
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 60
- If the ongoing reindex is consuming too many resources it is possible to throttle the running reindex :
- You can check the index write throughput in ES monitoring to determine a sensible throttle. Since it defaults to no throttling at all it's safe to just set some throttle and observe the impact
curl -XPOST "$CLUSTER_URL/_reindex/$TASK_ID/_rethrottle?requests_per_second=500
-
If reindexing task fails, it will automatically revert to the original index -
Ensure any updates that only went to Destination index are replayed against Source Cluster by searching the logs for the updates https://gitlab.com/gitlab-org/gitlab/-/blob/e8e2c02a6dbd486fa4214cb8183d428102dc1156/ee/app/services/elastic/process_bookkeeping_service.rb#L23 and triggering those updates again using ProcessBookkeepingService#track
as well as any updates that went through sidekiq workersElasticCommitIndexerWorker
,ElasticDeleteProjectWorker
.
-
Delete incomplete index gitlab-production-20221020-2340
by runningcurl -XDELETE "$CLUSTER_URL/gitlab-production-20221020-2340"
-
Set label changeaborted /label ~change::aborted
Monitoring
Key metrics to observe
- Metric: Elasticsearch cluster health
- Location: https://00a4ef3362214c44a044feaa539b4686.us-central1.gcp.cloud.es.io:9243/app/monitoring#/overview?_g=(cluster_uuid:HdF5sKvcT5WQHHyYR_EDcw)
- What changes to this metric should prompt a rollback: Unhealthy nodes/indices that do not recover
- Metric: Elasticsearch monitoring in Grafana
- Metric: Indexing queues
- Location: https://dashboards.gitlab.net/d/sidekiq-main/sidekiq-overview?orgId=1
- What changes to this metric should prompt a rollback: After unpausing the indexing is failing and the queues are constantly growing
Change Reviewer checklist
-
Check if the following applies: - The scheduled day and time of execution of the change is appropriate.
- The change plan is technically accurate.
- The change plan includes estimated timing values based on previous testing.
- The change plan includes a viable rollback plan.
- The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
Check if the following applies: - The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
- The change plan includes success measures for all steps/milestones during the execution.
- The change adequately minimizes risk within the environment/service.
- The performance implications of executing the change are well-understood and documented.
- The specified metrics/monitoring dashboards provide sufficient visibility for the change.
- If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
- The change has a primary and secondary SRE with knowledge of the details available during the change window.
- The labels blocks deployments and/or blocks feature-flags are applied as necessary
Change Technician checklist
-
Check if all items below are complete: - The change plan is technically accurate.
- This Change Issue is linked to the appropriate Issue and/or Epic
- Change has been tested in staging and results noted in a comment on this issue.
- A dry-run has been conducted and results noted in a comment on this issue.
- The change execution window respects the Production Change Lock periods.
- For C1 and C2 change issues, the change event is added to the GitLab Production calendar.
- For C1 and C2 change issues, the SRE on-call has been informed prior to change being rolled out. (In #production channel, mention
@sre-oncall
and this issue and await their acknowledgement.) - For C1 and C2 change issues, the SRE on-call provided approval with the eoc_approved label on the issue.
- For C1 and C2 change issues, the Infrastructure Manager provided approval with the manager_approved label on the issue.
- Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention
@release-managers
and this issue and await their acknowledgment.) - There are currently no active incidents that are severity1 or severity2
- If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.
Edited by Dmitry Gruzd