Skip to content

Draft: Segment notes advanced search index

John Mason requested to merge jm-user-segmented-index into master

What does this MR do and why?

Uses consistent hashing in order to segment indices for users. This MR introduces 36 segments. We could increase the number of segments and only have to reindex a small portion of data.

(better description coming soon)

Will move some of these steps to their own issues.

  • whenever a Segment is created in the DB, a worker is scheduled to 1) create empty index for segment, 2) create segment alias, 3) update global alias to include segment alias
  • a Segment can be "deactivated" by setting active: false. This should schedule a worker to remove segment alias from global alias. (no deletes for now)
  • introduces Elastic::IndexSetting#number_of_segments with a default number of 1.
  • reindexing task should create number_of_segments and corresponding aliases
  • migration to update a singular segment's alias to also point to global index. This will allow segments to work for SMC with legacy indices. gl-prod-users-s001 => gl-prod-users
  • cache strategy because refs are used frequently

Screenshots or screen recordings

image

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by John Mason

Merge request reports