Add support for indexed bytes/sec to code/wiki indexing SLI
Background
We currently measure the indexing SLI for global search by taking the search_indexing_duration_s
in the ElasticCommitIndexerWorker
for Code and Wiki. We recently enabled search_indexed_bytes_per_second
for Non-Code/Wiki data. This needs to be implemented for Code/Wiki now
Proposal
Measure the number of bytes (this may need to be done in the gitlab-elasticsearch-indexer and pass that information back to the ElasticCommitIndexerWorker
so it can be logged. This work must be done in a few MRs:
- Measure the number of bytes indexed when running the gitlab-elasticsearch-indexer and pass back to the caller.
- Change the indexer so it outputs one json with everything in it (it must contain the logs, bytes/sec statistics, and status code). The rails code can print the error instead of having to scan all the logs to find out what happened. The indexer should always be the json object (for success or error states)
- Introduce the new measurement in the Rails code and log it
- Get enough data (at least 7 days) from Kibana
- Replace existing apdex recording with the new measurement
- Continue work from #391488 (comment 1277012579) to add
number_of_commits
,repository_size
, andwiki_size
(if wiki is being indexed) to logging once this can be calculated and returned by indexer