Skip to content

Add index_and_search_validation elastic rake task

Terri Churequested to merge
tchu-add-index-and-search-check-as-rake-task into master

What does this MR do and why?

This merge request improves the Elasticsearch upgrade process in GitLab by adding better validation and clearer documentation.

The main changes include:

  1. Enhanced upgrade instructions: The documentation now provides clearer step-by-step guidance for upgrading Elasticsearch, including proper links to specific sections and a new validation step to ensure everything is working correctly after the upgrade.

  2. New validation tool: A new command (gitlab:elastic:index_and_search_validation) has been added that administrators can run to verify that their Elasticsearch cluster is healthy and functioning properly. This helps catch issues early before resuming normal operations.

  3. Better user interface documentation: New sections explain how to enable/disable Elasticsearch search and pause/resume indexing through GitLab's admin interface, making these operations more accessible to administrators.

  4. Improved logging: The validation service now accepts custom loggers, allowing better integration with different parts of the system and more flexible logging output.

  5. Comprehensive testing: Extensive test coverage has been added to ensure the new validation functionality works reliably.

These changes make Elasticsearch upgrades safer and more user-friendly by providing administrators with better tools to validate their setup and clearer instructions to follow during the upgrade process.

References

Screenshots or screen recordings

Before After

How to set up and validate locally

bundle exec rake gitlab:elastic:index_and_search_validation
  1. run the rake task (without elasticsearch setup for gdk if you'd like)
  2. setup gdk for elasticsearch
  3. run the rake task
 Running index and search validation...
 Index and search validation completed successfully
  1. set one of the indexes as read only
curl -X PUT "localhost:9200/gitlab-development-work_items/_settings" -H 'Content-Type: application/json' -d'
{
  "index.blocks.write": true
}
'
  1. run the rake task
Running index and search validation...
An error occurred while validating search cluster (error_message: [403] {"error":{"root_cause":[{"type":"cluster_block_exception","reason":"index [gitlab-development-work_items-20250801-1921] blocked by: [FORBIDDEN/8/index write (api)];"}],"type":"cluster_block_exception","reason":"index [gitlab-development-work_items-20250801-1921] blocked by: [FORBIDDEN/8/index write (api)];"},"status":403}, error_class: Elasticsearch::Transport::Transport::Errors::Forbidden)
Index and search validation failed
  1. shutdown elasticsearch in gdk gdk stop elasticsearch
  2. run the rake task
 Running index and search validation...
 The search cluster is unreachable
 Index and search validation failed

NOTE: remember to undo the write block

curl -X PUT "localhost:9200/gitlab-development-work_items/_settings" -H 'Content-Type: application/json' -d'
  {
    "index.blocks.write": false
  }
  '

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Terri Chu

Merge request reports