Merge Requests lazy_upvotes loads too many records

Summary

This bug was discovered during the root cause investigation for gitlab-com/gl-infra/production#5568 (closed)

Steps to reproduce

  • Queue up a large number of merge requests for indexing
  • Run the ElasticBulkCronWorker in the rails console ElasticBulkCronWorker.new.perform
  • look at the queries generated. There is a large query generated for AwardEmoji that has the potential to contain a large number of merge request ids (the way bulk indexing works is there are 16 redis queues with 1,000 max items per queue, so a total of 16,000 items are possible). We saw 6,200 in a production query

What is the current bug behavior?

The BatchLoader code is supposed to preload counts for all merge requests, but allows too many ids at once and the queries time out on production.

What is the expected correct behavior?

Queries should not time out.

Possible fixes

The BatchLoader queries should be further batched into smaller chunks to prevent query timeout on production.

Edited by Changzheng Liu