Skip to content

Add support for ES7 & Drop suport for ES5

Dylan Griffith requested to merge 118662-drop-support-es-v5-support-v7 into master

To be merged after we've released gitlab-elasticsearch-indexer 2.0.0

What does this MR do?

  1. Remove the ability to run against v5 by taking out the index.mapping.single_type logic
  2. Change our CI to test against 6.4.2 since we no longer want to support ES5
  3. Remove the standard token filter since it is not available in ES 7 and according to the docs it does nothing anyway
  4. Add support for ES7 by setting include_type_name when creating the index. This will ensure ES7 behaves like ES6 when creating mappings. See https://www.elastic.co/blog/moving-from-types-to-typeless-apis-in-elasticsearch-7-0 for more information.
  5. Monkey patch the elasticsearch-model gem so that it the Elasticsearch::Model::Response::Results#count method handles the new object format response from ES7 servers

Equivalent changes for the gitlab-elasticsearch-indexer at gitlab-elasticsearch-indexer!54 (merged)

NOTE: I ran CI that verifies we could test against Elasticsearch V7 https://gitlab.com/gitlab-org/gitlab/pipelines/109413089 . This commit was just to test and was reverted for now until we implement #196503 (comment 270297726)

NOTE: Another incompatible change that has happened in ES7 is that Elasticsearch server only counts the first 10,000 results in the total count by default which means that until we make some changes in GitLab we will not be able to paginate past 10,000 results. This seems like we can live with this limitation for now so I've extracted an issue to deal with this later in #196627 (closed)

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Manual QA

6.5.1

  1. Enable ES
  2. Create a project with snippets, code, commits, issues, issue comments, merge request comments
  3. Perform searches for all resource types using * search
  4. Perform boolean search tests like a | b and (a | b) + c
  5. Trigger a complete re-index using bundle exec rake gitlab:elastic:index and again try all searches
  6. Start again but with only having Elasticsearch enabled for a single group and again try all searches
  7. Check the elasticsearch/logs/*.log and gdk tail elasticsearch for any errors or warnings

7.5.1

  1. Enable ES
  2. Create a project with snippets, code, commits, issues, issue comments, merge request comments
  3. Perform searches for all resource types using * search
  4. Perform boolean search tests like a | b and (a | b) + c
  5. Trigger a complete re-index using bundle exec rake gitlab:elastic:index and again try all searches
  6. Start again but with only having Elasticsearch enabled for a single group and again try all searches
  7. Check the elasticsearch/logs/*.log and gdk tail elasticsearch for any errors or warnings

#118662 (closed) #11197 (closed) #12641 (closed)

Edited by 🤖 GitLab Bot 🤖

Merge request reports