Advanced search we can package into omnibus
Description
GitLab EE has Advanced code search, global search, and advanced search syntax, all powered by an external Elasticsearch server. This means it's disabled by default, because Elasticsearch is written in Java and we don't package Java applications into omnibus. As a result, CE and non-elastic EE search lacks features and performance.
Although elasticsearch is very standard, it is difficult to set up and run, so practically, many of our users and customers are stuck with this default search. Our customers who can set up and maintain their own elasticsearch cluster benefit greatly from doing so, but the feature is more fragile than it ought to be as a result of the external integration.
Proposal
I suggest we retain elasticsearch support for EEP, but add an additional search driver to CE+EE, based on an indexing+search server written in Go, using blevesearch: http://www.blevesearch.com . This won't be intended to scale to GitLab.com scale, but instead, to handle the much more common case where everything will fit on a single server.
A great deal of the code we need to do this is already present in EE - the presence of elasticsearch means that all the necessary triggers and pluggable-search-backend points are present there. So the majority of the effort goes into:
- Porting non-elasticsearch-specific infrastructure to CE
- Building the search server component
The first doesn't hurt the elasticsearch feature in any way, since you still wouldn't be able to use elasticsearch with CE. It also reduces the diff between CE and EE, which is of value all by itself.
Bleve gives us all the necessary components to do the second, and we could tackle it incrementally - enabling global commit search, then global code search, and so on over time. "Advanced search syntax" would come last, and might differ significantly from the elasticsearch variant - but I've not done more than cursory checking of that.
It's certainly not a trivial operation, however, and given we do already have elasticsearch, it might be hard to justify spending GitLab time on it.
So, I've put this issue into CE. I think we could mark this ~"Accepting Merge Requests" and get community contributions to build this up to complete over time. It would make all flavours of GitLab more attractive for small installations, with elasticsearch integration continuing to provide compelling benefits on EE for larger installations.
Links / references
/cc @victorwu @smcgivern