Skip to content

Refactor reindex service log to include class name and use key/values

Terri Chu requested to merge tchu-refactor-cluster-reindexing-service-logging into master

What does this MR do and why?

Found while debugging #425195 (closed)

I was debugging a failure in staging today and had a difficult time finding failures from ElasticClusterReindexingCronWorker. That worker executes the Elastic::ClusterReindexingService. The logger in that service is not setting the class name. I opened this MR to refactor the logging to:

  • use structured logging in the payload and key/value pairs for values
  • use GitlabLoggable which populates the class_name field
  • add a missing failures case for errors from Elasticsearch.
  • update specs

Screenshots or screen recordings

N/A, only backend changes

How to set up and validate locally

  1. enable elasticsearch in gdk
  2. Setup elasticsearch
  3. Kick off zero-downtime reindexing via the UI: http://gdk.test:3000/admin/application_settings/advanced_search
  4. open rails console and run the service manually (or you will have to wait 10 minutes for the cron worker): ::Elastic::ClusterReindexingService.new.execute
  5. watch the elasticsearch.log file to see the logging updates
  6. verify the logs contain class and all logging parameters are passed as key/value pairs vs. being in the message
  7. verify that the reindexing process completes successfully, check Elastic::ReindexingTask.current in the console. if it's empty, then the process it complete. you will need to run the ::Elastic::ClusterReindexingService.new.execute service multiple times in rails console to complete the reindexing process.

example logs:

{"severity":"INFO","time":"2023-09-14T18:01:13.223Z","class":"Elastic::ClusterReindexingService","message":"Reindex task started","task_id":"99YgTgMFRSC38-gIr1XM-A:181045","index_from":"gitlab-development-issues-20230823-1719-reindex-3-0","index_to":"gitlab-development-issues-20230914-1800-reindex-4-0","slice":4}
{"severity":"INFO","time":"2023-09-14T18:01:13.238Z","class":"Elastic::ClusterReindexingService","message":"Reindex task started","task_id":"99YgTgMFRSC38-gIr1XM-A:181057","index_from":"gitlab-development-issues-20230823-1719-reindex-3-0","index_to":"gitlab-development-issues-20230914-1800-reindex-4-0","slice":5}
{"severity":"INFO","time":"2023-09-14T18:01:13.247Z","class":"Elastic::ClusterReindexingService","message":"Reindex task started","task_id":"99YgTgMFRSC38-gIr1XM-A:181067","index_from":"gitlab-development-issues-20230823-1719-reindex-3-0","index_to":"gitlab-development-issues-20230914-1800-reindex-4-0","slice":6}
{"severity":"INFO","time":"2023-09-14T18:01:13.257Z","class":"Elastic::ClusterReindexingService","message":"Reindex task started","task_id":"99YgTgMFRSC38-gIr1XM-A:181077","index_from":"gitlab-development-issues-20230823-1719-reindex-3-0","index_to":"gitlab-development-issues-20230914-1800-reindex-4-0","slice":7}
{"severity":"INFO","time":"2023-09-14T18:01:13.266Z","class":"Elastic::ClusterReindexingService","message":"Reindex task started","task_id":"99YgTgMFRSC38-gIr1XM-A:181087","index_from":"gitlab-development-issues-20230823-1719-reindex-3-0","index_to":"gitlab-development-issues-20230914-1800-reindex-4-0","slice":8}
{"severity":"INFO","time":"2023-09-14T18:01:13.318Z","class":"Elastic::ClusterReindexingService","message":"Reindex task started","task_id":"99YgTgMFRSC38-gIr1XM-A:181097","index_from":"gitlab-development-issues-20230823-1719-reindex-3-0","index_to":"gitlab-development-issues-20230914-1800-reindex-4-0","slice":9}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Terri Chu

Merge request reports