Advanced Search: Problem with 1-step mapping update and backfill

While working on !70042 (merged) I noticed a problem with the AddUpvotesToMergeRequests migration and with the 1-step approach of adding mappings & backfill:

1. GitLab is updated
2. Sidekiq nodes pick up the new upvotes field right away since it doesn't have if Elastic::DataMigrationService.migration_has_finished?(:add_upvotes_to_merge_requests) condition
3. All the updates for merge_requests fail with strict_dynamic_mapping_exception since we have "mappings": { "dynamic": "strict" } set.
4. This only stops when the migration runs the first time and mapping is updated, which might take up to 30 minutes if all migrations are up to date and more if there are other uncompleted migrations.

click to see failure

curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/gitlab-test-merge_requests/_doc' -d '{
 "id": 440,
 "iid": 1,
 "target_branch": "my-branch-2",
 "source_branch": "my-branch-1",
 "title": "My title 1",
 "description": null,
 "created_at": "2021-09-17T10:35:24.351Z",
 "updated_at": "2021-09-17T10:35:24.351Z",
 "state": "opened",
 "merge_status": "can_be_merged",
 "source_project_id": 894,
 "target_project_id": 894,
 "project_id": 894,
 "author_id": 906,
 "visibility_level": 0,
 "merge_requests_access_level": 20,
 "upvotes": 0,
 "type": "merge_request"
}'

{"error":{"root_cause":[{"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [upvotes] within [doc] is not allowed"}],"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [upvotes] within [doc] is not allowed"},"status":400}

---

Proposed fix for AddUpvotesToMergeRequests

For the AddUpvotesToMergeRequests migration, I think we should create another migration only for adding upvotes mapping and set its version before AddUpvotesToMergeRequests and guard the upvotes field with Elastic::DataMigrationService.migration_has_finished?(:...).

General solution

I believe we should have 2 different helpers. One for adding mappings and another for the backfill and should always check if the mapping migration is completed in the as_indexed_json methods.