Get ElasticSearch in shape

ElasticSearch has been a tricky subject over the past months and we need to bring it in shape. Roughly speaking we would need the following (in no particular order):

  • Better logging, we should be able to see where the indexing process is, what it's indexing, etc
  • Better performance monitoring (as much code should be instrumented as possible)
  • Better performing code
  • Better infrastructure monitoring (unless this is already taken care of), somewhat similar to the monitoring for PostgreSQL (slow queries, etc)

The 3rd item is the hardest and most time consuming part. The first step towards this process is to note down the current performance problems with the various bits of code, and create issues for this in GitLab EE (since ElasticSearch is EE only). We can then assign these issues to those who'd like to work on it.

@pacoguzman @eReGeBe @maratkalibek @ahmadsherif : In today's call @maratkalibek volunteered to help out on this, but I'd like to have another performance engineer work specifically on the code part. Unless somebody volunteers for this part I'll assign somebody randomly using the power of Ruby.