Advanced Search - Elasticsearch 7.7.0 x_content_parse_exception error
Summary
When using Elasticsearch 7.7.0 with GitLab 12.10.x for advanced search, you get a 500 error when searching for issues if more than one issue has the same word in it. I was unable to recreate this issue using 7.6.0.
Steps to reproduce
- Connect GitLab 12.10.x to ES 7.7.0 and enable advanced search.
- Create a project and create an issue with the word
layer
in the description and title. (I usedThis is a layer test 1
in both fields) - Search for the word
layer
in the UI (this works) - Create another issue with the word
layer
in the description and title. (I usedThis is a layer test 2
in both fields) - Search for the word
layer
in the UI and click on Issues.
What is the current bug behavior?
When searching for the word, a 500 error is shown in the UI and an error is thrown in the production.log
.
What is the expected correct behavior?
Search works as it is suppose to.
Relevant logs and/or screenshots
gitlab | Elasticsearch::Transport::Transport::Errors::BadRequest ([400] {"error":{"root_cause":[{"type":"x_content_parse_exception","reason":"[1:240] [bool] failed to parse field [should]"}],"type":"x_content_parse_exception","reason":"[1:240] [bool] failed to parse field [filter]","caused_by":{"type":"x_content_parse_exception","reason":"[1:240] [bool] failed to parse field [should]","caused_by":{"type":"illegal_state_exception","reason":"expected value but got [START_ARRAY]"}}},"status":400}):
gitlab |
gitlab | config/initializers/elastic_client_setup.rb:36:in `total'
gitlab | ee/lib/gitlab/elastic/search_results.rb:104:in `merge_requests_count'
gitlab | lib/gitlab/metrics/instrumentation.rb:161:in `block in merge_requests_count'
gitlab | lib/gitlab/metrics/method_call.rb:36:in `measure'
gitlab | lib/gitlab/metrics/instrumentation.rb:161:in `merge_requests_count'
gitlab | ee/lib/gitlab/elastic/search_results.rb:71:in `formatted_count'
gitlab | lib/gitlab/metrics/instrumentation.rb:161:in `block in formatted_count'
gitlab | lib/gitlab/metrics/method_call.rb:36:in `measure'
gitlab | lib/gitlab/metrics/instrumentation.rb:161:in `formatted_count'
gitlab | app/controllers/search_controller.rb:46:in `count'
gitlab | app/controllers/application_controller.rb:564:in `block in allow_gitaly_ref_name_caching'
gitlab | lib/gitlab/gitaly_client.rb:350:in `allow_ref_name_caching'
gitlab | app/controllers/application_controller.rb:563:in `allow_gitaly_ref_name_caching'
gitlab | ee/lib/gitlab/ip_address_state.rb:10:in `with'
gitlab | ee/app/controllers/ee/application_controller.rb:43:in `set_current_ip_address'
gitlab | app/controllers/application_controller.rb:479:in `set_current_admin'
gitlab | lib/gitlab/session.rb:11:in `with_session'
gitlab | app/controllers/application_controller.rb:470:in `set_session_storage'
gitlab | lib/gitlab/i18n.rb:55:in `with_locale'
gitlab | lib/gitlab/i18n.rb:61:in `with_user_locale'
gitlab | app/controllers/application_controller.rb:464:in `set_locale'
gitlab | lib/gitlab/error_tracking.rb:48:in `with_context'
gitlab | app/controllers/application_controller.rb:559:in `sentry_context'
gitlab | lib/gitlab/application_context.rb:52:in `block in use'
gitlab | lib/gitlab/application_context.rb:52:in `use'
gitlab | lib/gitlab/application_context.rb:20:in `with_context'
gitlab | app/controllers/application_controller.rb:455:in `set_current_context'
gitlab | lib/gitlab/middleware/rails_queue_duration.rb:29:in `call'
gitlab | lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
gitlab | lib/gitlab/metrics/transaction.rb:62:in `run'
gitlab | lib/gitlab/metrics/rack_middleware.rb:17:in `call'
gitlab | lib/gitlab/request_profiler/middleware.rb:17:in `call'
gitlab | ee/lib/gitlab/jira/middleware.rb:19:in `call'
gitlab | lib/gitlab/middleware/go.rb:20:in `call'
gitlab | lib/gitlab/etag_caching/middleware.rb:13:in `call'
gitlab | lib/gitlab/middleware/multipart.rb:124:in `call'
gitlab | lib/gitlab/middleware/read_only/controller.rb:53:in `call'
gitlab | lib/gitlab/middleware/read_only.rb:18:in `call'
gitlab | lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
gitlab | lib/gitlab/middleware/basic_health_check.rb:25:in `call'
gitlab | lib/gitlab/middleware/request_context.rb:23:in `call'
gitlab | config/initializers/fix_local_cache_middleware.rb:9:in `call'
gitlab | lib/gitlab/metrics/requests_rack_middleware.rb:60:in `call'
gitlab | lib/gitlab/middleware/release_env.rb:12:in `call'
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
``` root@172:/# gitlab-rake gitlab:env:info System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.7 Git Version: 2.26.2 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 12.10.6-ee Revision: 933c215f042 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 11.7 URL: http://172.19.0.3 HTTP Clone URL: http://172.19.0.3/some-group/some-project.git SSH Clone URL: git@172.19.0.3:some-group/some-project.git Elasticsearch: yes Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 12.2.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git ```
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)root@172:/# gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 12.2.0 ? ... OK (12.2.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1/1 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.26.2) Git user has default SSH configuration? ... yes Active users: ... 1 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... yes (7.7.0) Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
Possible fixes
The current workaround is to use Elasticsearch 7.6.0 as I was unable to recreate this issue with it. I was unable to determine the exact reason why this is happening, but looking at Elastic's Release Notes for 7.7.0, there were changes related to XContent
.