Advanced Search - Elasticsearch 7.7.0 x_content_parse_exception error

Summary

When using Elasticsearch 7.7.0 with GitLab 12.10.x for advanced search, you get a 500 error when searching for issues if more than one issue has the same word in it. I was unable to recreate this issue using 7.6.0.

Steps to reproduce

Connect GitLab 12.10.x to ES 7.7.0 and enable advanced search.
Create a project and create an issue with the word layer in the description and title. (I used This is a layer test 1 in both fields)
Search for the word layer in the UI (this works)
Create another issue with the word layer in the description and title. (I used This is a layer test 2 in both fields)
Search for the word layer in the UI and click on Issues.

What is the current bug behavior?

When searching for the word, a 500 error is shown in the UI and an error is thrown in the production.log.

What is the expected correct behavior?

Search works as it is suppose to.

Relevant logs and/or screenshots

gitlab           | Elasticsearch::Transport::Transport::Errors::BadRequest ([400] {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:240] [bool] failed to parse field [should]"}],"type":"x_content_parse_exception","reason":"[1:240] [bool] failed to parse field [filter]","caused_by":{"type":"x_content_parse_exception","reason":"[1:240] [bool] failed to parse field [should]","caused_by":{"type":"illegal_state_exception","reason":"expected value but got [START_ARRAY]"}}},"status":400}):
gitlab           |   
gitlab           | config/initializers/elastic_client_setup.rb:36:in `total'
gitlab           | ee/lib/gitlab/elastic/search_results.rb:104:in `merge_requests_count'
gitlab           | lib/gitlab/metrics/instrumentation.rb:161:in `block in merge_requests_count'
gitlab           | lib/gitlab/metrics/method_call.rb:36:in `measure'
gitlab           | lib/gitlab/metrics/instrumentation.rb:161:in `merge_requests_count'
gitlab           | ee/lib/gitlab/elastic/search_results.rb:71:in `formatted_count'
gitlab           | lib/gitlab/metrics/instrumentation.rb:161:in `block in formatted_count'
gitlab           | lib/gitlab/metrics/method_call.rb:36:in `measure'
gitlab           | lib/gitlab/metrics/instrumentation.rb:161:in `formatted_count'
gitlab           | app/controllers/search_controller.rb:46:in `count'
gitlab           | app/controllers/application_controller.rb:564:in `block in allow_gitaly_ref_name_caching'
gitlab           | lib/gitlab/gitaly_client.rb:350:in `allow_ref_name_caching'
gitlab           | app/controllers/application_controller.rb:563:in `allow_gitaly_ref_name_caching'
gitlab           | ee/lib/gitlab/ip_address_state.rb:10:in `with'
gitlab           | ee/app/controllers/ee/application_controller.rb:43:in `set_current_ip_address'
gitlab           | app/controllers/application_controller.rb:479:in `set_current_admin'
gitlab           | lib/gitlab/session.rb:11:in `with_session'
gitlab           | app/controllers/application_controller.rb:470:in `set_session_storage'
gitlab           | lib/gitlab/i18n.rb:55:in `with_locale'
gitlab           | lib/gitlab/i18n.rb:61:in `with_user_locale'
gitlab           | app/controllers/application_controller.rb:464:in `set_locale'
gitlab           | lib/gitlab/error_tracking.rb:48:in `with_context'
gitlab           | app/controllers/application_controller.rb:559:in `sentry_context'
gitlab           | lib/gitlab/application_context.rb:52:in `block in use'
gitlab           | lib/gitlab/application_context.rb:52:in `use'
gitlab           | lib/gitlab/application_context.rb:20:in `with_context'
gitlab           | app/controllers/application_controller.rb:455:in `set_current_context'
gitlab           | lib/gitlab/middleware/rails_queue_duration.rb:29:in `call'
gitlab           | lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
gitlab           | lib/gitlab/metrics/transaction.rb:62:in `run'
gitlab           | lib/gitlab/metrics/rack_middleware.rb:17:in `call'
gitlab           | lib/gitlab/request_profiler/middleware.rb:17:in `call'
gitlab           | ee/lib/gitlab/jira/middleware.rb:19:in `call'
gitlab           | lib/gitlab/middleware/go.rb:20:in `call'
gitlab           | lib/gitlab/etag_caching/middleware.rb:13:in `call'
gitlab           | lib/gitlab/middleware/multipart.rb:124:in `call'
gitlab           | lib/gitlab/middleware/read_only/controller.rb:53:in `call'
gitlab           | lib/gitlab/middleware/read_only.rb:18:in `call'
gitlab           | lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
gitlab           | lib/gitlab/middleware/basic_health_check.rb:25:in `call'
gitlab           | lib/gitlab/middleware/request_context.rb:23:in `call'
gitlab           | config/initializers/fix_local_cache_middleware.rb:9:in `call'
gitlab           | lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'
gitlab           | lib/gitlab/middleware/release_env.rb:12:in `call'

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info


```
root@172:/# gitlab-rake gitlab:env:info

System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.5p114
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.7
Git Version:	2.26.2
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	12.10.6-ee
Revision:	933c215f042
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		http://172.19.0.3
HTTP Clone URL:	http://172.19.0.3/some-group/some-project.git
SSH Clone URL:	git@172.19.0.3:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	12.2.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git
```

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:check SANITIZE=true)

root@172:/# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 12.2.0 ? ... OK (12.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
1/1 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.5)
Git version >= 2.22.0 ? ... yes (2.26.2)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
Elasticsearch version 5.6 - 6.x? ... yes (7.7.0)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

The current workaround is to use Elasticsearch 7.6.0 as I was unable to recreate this issue with it. I was unable to determine the exact reason why this is happening, but looking at Elastic's Release Notes for 7.7.0, there were changes related to XContent.

Edited May 18, 2020 by Eldridge Henley