Add support for ES7 & Drop suport for ES5
To be merged after we've released gitlab-elasticsearch-indexer 2.0.0
What does this MR do?
- Remove the ability to run against v5 by taking out the
index.mapping.single_typelogic - Change our CI to test against
6.4.2since we no longer want to support ES5 - Remove the standard token filter since it is not available in ES 7 and according to the docs it does nothing anyway
- Add support for ES7 by setting
include_type_namewhen creating the index. This will ensure ES7 behaves like ES6 when creating mappings. See https://www.elastic.co/blog/moving-from-types-to-typeless-apis-in-elasticsearch-7-0 for more information. - Monkey patch the
elasticsearch-modelgem so that it theElasticsearch::Model::Response::Results#countmethod handles the new object format response from ES7 servers
Equivalent changes for the gitlab-elasticsearch-indexer at gitlab-elasticsearch-indexer!54 (merged)
NOTE: I ran CI that verifies we could test against Elasticsearch V7 https://gitlab.com/gitlab-org/gitlab/pipelines/109413089 . This commit was just to test and was reverted for now until we implement #196503 (comment 270297726)
NOTE: Another incompatible change that has happened in ES7 is that Elasticsearch server only counts the first 10,000 results in the total count by default which means that until we make some changes in GitLab we will not be able to paginate past 10,000 results. This seems like we can live with this limitation for now so I've extracted an issue to deal with this later in #196627 (closed)
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec -
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Manual QA
6.5.1
-
Enable ES -
Create a project with snippets, code, commits, issues, issue comments, merge request comments -
Perform searches for all resource types using *search -
Perform boolean search tests like a | band(a | b) + c -
Trigger a complete re-index using bundle exec rake gitlab:elastic:indexand again try all searches -
Start again but with only having Elasticsearch enabled for a single group and again try all searches -
Check the elasticsearch/logs/*.logandgdk tail elasticsearchfor any errors or warnings
7.5.1
-
Enable ES -
Create a project with snippets, code, commits, issues, issue comments, merge request comments -
Perform searches for all resource types using *search -
Perform boolean search tests like a | band(a | b) + c -
Trigger a complete re-index using bundle exec rake gitlab:elastic:indexand again try all searches -
Start again but with only having Elasticsearch enabled for a single group and again try all searches -
Check the elasticsearch/logs/*.logandgdk tail elasticsearchfor any errors or warnings