Address schema version ordering complexities for parallel development on an advanced search index

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Issue

We currently have sequencing complexities for schema_version on an advanced search index, when we have multiple fields being newly introduced on the index with parallel development across teams.

Example code snippet.

We are facing the below hardships:

• The complexity increases when the ES backfill on a field is dependent on a PostGres BBM and we have more ES migrations lined up.

• Observing a situation where every migration before merge needs careful thought through and requires necessary refactoring by changing the migration timestamp, modifying the schema version ordering etc on the reference class. Example thread.

• For ES backfills which are dependent on PG BBM, we also have to consider the Dedicated and SM instances which complicates the ordering logic of schema_version furthur and we have to divide the migration plan into multiple MR's for gitlab.com, Dedicated and SM.

• The ordering logic furthur adds complications if an older field has a regression.

• The ordering logic add complication when a field value assignment is based on Feature Flags.

Expectation:

Ideally the framework should seamlessly handle the schema version assignment instead of relying on conditional code checks on the reference class.