Elasticsearch administrators currently have no information about what information has been added to the Index or what the overall status of the Index is. This makes it hard to troubleshoot issues or understand if there are problems with the indexing process.
Intended users
GitLab Instance admins for Elasticsearch
Further details
Proposal
There should be information available to Elasticsearch administrators that informs of the current status of the Index. This should be supported at a high level (XX% Complete) as well as on an individual component level (wiki, project, snippets, etc...).
Permissions and Security
This information should only be available to GitLab instance administrators.
Documentation
Elasticsearch administration documentation should be updated.
Links / references
Old Description
We had a customer attempt to upgrade GitLab and reindex their Elasticsearch database. They had an issue where no search results turned up, but a quick `curl` test seemed to suggest Elasticsearch did have some data.
Right now it's really difficult to get a high-level view of how many projects, issues, merge requests, etc. have been indexed. Is there a way we can add something to the admin page that shows this data? Ideally, there would also a way to search for a term, see the query it sends, and the results it gets back from Elasticsearch.
I was thinking last night - it would be neat if GitLab had a button to reveal the raw query right in the user search UI. Kibana does this I think. It would allow us to grab the raw query and issue it directly for testing purposes.
This is a pretty hard problem if you want the percentage of how far along a project is, because we don't currently have this info anywhere but inside the indexer, and it doesn't communicate back to us.
We could probably start by iterating on it - step one would be getting which Projects are indexed (and "are indexed" is kind of iffy, since we'd be going with the existence of an IndexStatus object in the database
@DouweM I think that's reasonable for fairly basis status, especially if we're mostly focused on the project and wiki repository at this point.
In lieu of us tracking status about the database indexing, we could include information about how many documents of each type are present in the elasticsearch index for a given project - that would at least allow us to see "there are no documents for this project", vs. "there are lots of documents for this project", which would have been useful for me this week!
@phikai Scheduled a first meeting with @nick.thomas tomorrow to understand context and potential user stories and tasks, should have a better understanding after that.
Kai Armstrongchanged title from Add an admin screen to check status of Elasticsearch indexing to Instance Administrators should be able to check status of Elasticsearch Index
changed title from Add an admin screen to check status of Elasticsearch indexing to Instance Administrators should be able to check status of Elasticsearch Index
@dsatcher Is there backend work that doesn't also need UI work that could be done here, or is all of that a throwaway as well until more of the UI is complete.
I think the idea of this initially was to create some of this and then implement in the UI design when we get there, but if that needs to happen more synchronously we can move it back.