Elasticsearch v6 - migrate from parent/child relationships to joins

What does this MR do?

Since ES6 only allows one mapping type per index and we need to have documents in a single index in order to use join datatypes, I had to make quite a few changes to how we deal with elasticsearch in the codebase:

• I changed all our document types to be doc
• In order to still be able to query per type, I implemented a custom type field in our elasticsearch documents, called simply type
• The join type was created granularly, that is to say, it has a mapping per type: project->issue, project->blob, etc.
• We now use an es_id for elasticsearch's _id field. This is comprised of the model class name and the ID in the database
• Because of the es_id change a lot of changes were required to the elasticsearch-model gem. These changes have been encapsulated in the ee/lib/gem_extensions folder and are injected inside the elasticsearch client initializer.

Breaking changes

We can no longer support Elasticsearch versions under 5.6. This is because the join field is only supported from versions 5.6 and upwards.

Wiki searches now must use a new type: wiki_blob. I've edited existing usages in the codebase but this is good to keep in mind

What are the relevant issue numbers?

#4218 (closed)

Does this MR meet the acceptance criteria?

• Changelog entry added, if necessary
• Documentation created/updated
• Tests added for this feature/bug
• Conforms to the code review guidelines
• Conforms to the merge request performance guidelines
• Conforms to the style guides
• Conforms to the database guides
• EE specific content should be in the top level /ee folder
• For a paid feature, have we considered GitLab.com plans, how it works for groups, and is there a design for promoting it to users who aren't on the correct plan?

Closes #4218 (closed)

	1 Error
	b2303efe: The commit subject must contain at least three words

	1 Warning
	This merge request is quite big (more than 1019 lines changed), please consider splitting it into multiple merge requests.

	1 Message
	This merge request adds or changes files that require a review from the docs team.

Docs Review

The following files require a review from the Documentation team:

• doc/development/elasticsearch.md
• doc/integration/elasticsearch.md

To make sure these changes are reviewed, mention @gl-docsteam in a separate comment, and explain what needs to be reviewed by the team. Please don't mention the team until your changes are ready for review.

Commit message standards

One or more commit messages do not meet our Git commit message standards. For more information on how to write a good commit message, take a look at How to Write a Git Commit Message.

Here is an example of a good commit message:

Reject ruby interpolation in externalized strings

When using ruby interpolation in externalized strings, they can't be
detected. Which means they will never be presented to be translated.

To mix variables into translations we need to use `sprintf`
instead.

Instead of:

    _("Hello #{subject}")

Use:

    _("Hello %{subject}") % { subject: 'world' }

This is an example of a bad commit message:

updated README.md

This commit message is bad because although it tells us that README.md is updated, it doesn't tell us why or how it was updated.

Generated by Danger

added 1 commit

• a1f16a01 - Clean up elasticsearch-model overrides

Compare with previous version

added 1 commit

• bb3e061b - Clean up elasticsearch-model overrides

Compare with previous version

added 1 commit

• a84681ec - Use elasticsearch 5.6 in CI tests. We cannot support ES5.5 and under with ES JOIN

Compare with previous version

added 1 commit

• 4f1a443a - ElasticSearch 6 - clean up specs

Compare with previous version

changed the description

added 1 commit

• 65e9fff0 - Elasticsearch - `es_type` in base class and fix namespace resolution issue

Compare with previous version

added 1 commit

• 9267cded - Elasticsearch - clean up ES mappings

Compare with previous version

added 300 commits

• 9267cded...25c8d7ce - 298 commits from branch master
• 34ddd589 - Elasticsearch v6 support
• 4bd2404a - Update elasticsearch docs to reflect v6 support

Compare with previous version

added 2 commits

• db235922 - Elasticsearch v6 support
• 7c5b7cf2 - Update elasticsearch docs to reflect v6 support

Compare with previous version

added 2 commits

• b113c903 - Elasticsearch v6 support
• 0be7c627 - Update elasticsearch docs to reflect v6 support

Compare with previous version

added 2 commits

• c2d65a88 - Elasticsearch v6 support
• c5230573 - Update elasticsearch docs to reflect v6 support

Compare with previous version

marked the checklist item Changelog entry added, if necessary as completed

marked the checklist item Tests added for this feature/bug as completed

marked the checklist item Documentation created/updated as completed

added documentation label

added 2 commits

• 25b45f55 - Elasticsearch v6 support
• fbe09db1 - Update elasticsearch docs to reflect v6 support

Compare with previous version

@nick.thomas could u bother you for a review here please? This was tested against ES5.6 and 6.4

Also ccing @smcgivern in case he wants to take an early look at this :)

FYI I still need to create a CE branch to backport some changes there was only one change which I removed. no need for a backport anymore

assigned to @nick.thomas

If @vsizov has time, he might want to review too.

Oh yes I'd love for @vsizov to give this a look as well if he has time! This is a big change, the more eyes the better

I will try to look into it tomorrow guys.

As a follow-up to this issue I propose we should comb through all the elasticsearch-related tests and add the :elastic tag to them so we can add some jobs to the pipeline to test against elasticsearch 5.6.12, since there ARE some differences between 5.6 and 6.4 and we wouldn't want to cause a regression

added 1 commit

• 412d7341 - Elasticsearch - tag specs that use it with :elastic

Compare with previous version

I did a quick look, I have no time this week for thorough review :(

• The changes look reasonable to me(in a context of iterative changes, I think, we don't have to use ES gems anymore as I already said. I believe you estimated the cost of the full removal of the elasticsearch-rails gem and you consider it to be more expensive, right?
• I like that schema is in one place, it's way easier to maintain now.
• Don't forget about existing customers who have to reindex everything. It should be noted everywhere: update guides, blog post and so on.
• Why can't we still use {entity}_id notion?

Thanks a ton for looking @vsizov!

I believe you estimated the cost of the full removal of the elasticsearch-rails gem and you consider it to be more expensive, right?

That's correct. In order to remove the gem we need to build our own modules that know how to retrieve ActiveRecord objects from the Elasticsearch results. That was the main reason why I went with monkey-patches instead of fully ripping it out

Why can't we still use {entity}_id notion?

Sorry, I'm not sure what you mean by that? We switched to using {entity}_id for elasticsearch _ids

added 1 commit

• 70ee09fa - IssueAssignee - extract EE class

Compare with previous version

added 1 commit

• 5e7066a6 - IssueAssignee - extract EE class

Compare with previous version

added 1 commit

• 96a6a35d - IssueAssignee - extract EE class

Compare with previous version

added 3 commits

• ba7424e0 - Elasticsearch v6 support
• 2a5b828a - Update elasticsearch docs to reflect v6 support
• 7f56345d - IssueAssignee - extract EE class

Compare with previous version