Skip to content

Upgrade to 18.1. causes strict_dynamic_mapping_exception for Elasticsearch version 7

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

When the initial index is created on a 17.4 instance and the instance is upgraded per our upgrade path, when you reach 18.1 version, ElasticIndexInitialBulkCronWorker throws indexing errors like below:

{"severity":"WARN","time":"2025-08-27T22:22:06.853Z","meta.caller_id":"ElasticIndexInitialBulkCronWorker","correlation_id":"2e7a4f4f3690641d0482f24230840178","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","meta.client_id":"ip/","message":"bulk_error","item":{"update":{"_index":"gitlab-production-work_items-20250826-2122","_type":"_doc","_id":"12","status":400,"error":{"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [routing] within [_doc] is not allowed"}}}}

@terrichu has done some initial investigation:

  • MR to introduce routing to base mapping for work items index <-- support for newly created indices
  • MR to migrate existing indices to add routing to mapping for work items index <-- support for existing indices
  • migration has a skip condition skip_if -> { !elasticsearch_8_plus? && !opensearch? }
  • what this means is that indexes that report as Elasticsearch 7.X will not have the new mapping added

Question / TBD: Per above, we expected to run into this issue post 17.4; but couldn't replicate it until upgrading to 18.1

Steps to reproduce

  1. Install GitLab version 17.4 and configure Elasticsearch (version 7)
  2. Create index from scratch (with some work item data)
  3. Upgrade to 18.1 following the upgrade path and finish migrations
  4. Change work item data and see if you get indexing errors

What is the current bug behavior?

ElasticIndexInitialBulkCronWorker errors with strict_dynamic_mapping_exception.

What is the expected correct behavior?

ElasticIndexInitialBulkCronWorker should not error.

Relevant logs and/or screenshots

{"severity":"WARN","time":"2025-08-27T22:22:06.853Z","meta.caller_id":"ElasticIndexInitialBulkCronWorker","correlation_id":"2e7a4f4f3690641d0482f24230840178","meta.root_caller_id":"Cronjob","meta.feature_category":"global_search","meta.client_id":"ip/","message":"bulk_error","item":{"update":{"_index":"gitlab-production-work_items-20250826-2122","_type":"_doc","_id":"12","status":400,"error":{"type":"strict_dynamic_mapping_exception","reason":"mapping set to strict, dynamic introduction of [routing] within [_doc] is not allowed"}}}}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Possible fixes

Option 1:

Add routing to the properties:

curl --request PUT \
  --url <ELATICSEARCH_URL>/gitlab-production-work_items/_mapping \
  --header 'Content-Type: application/json' \
  --data '{
  "properties": {
    "routing": {
      "type": "text"
    }
	}
}'

Option 2:

Do a full reindex.

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.

Edited by 🤖 GitLab Bot 🤖