Refactor search indexed bytes/sec to account for parallel processing
Background
!110936 (merged) introduced bulk cron worker parallel processing. While this sped up the rate at which the queue is drained it has impacted the search_indexed_bytes_per_second
calculation accuracy.
Proposal
The bytes / second calculation should now be done in the bulk_cron_worker
as it's running multiple shards at once. High level plan to address:
- store a start_time, end_time, and total bytes processed (need to determine where this can be stored and how)
- in bulk_cron_worker, if
shard_number
is not sent toperform
method, sum up all stored processing info and report an average