Reactive caching can lead to an explosion of Sidekiq jobs
https://log.gitlab.net/goto/f5914517fde047c6229da4e8e649285e shows that a single request to the merge request widget JSON generated 20,000+ log lines in Sidekiq for reactive caching jobs. These jobs are trying to get the CI status for the MR from an external Drone CI install.
Looking at the jobs in more detail, I see https://log.gitlab.net/goto/122da6a2c3146e9fc8fbe545b3030f6d. We can see that the arguments are always the same (with different JIDs), which means we're just constantly rescheduling jobs - but this is the opposite of the point of reactive caching, where we get the result once and then store it for a time. These jobs are completing at the rate of 1,000+ per hour, but running more frequently.
I'm marking this as ~Verify because it's to do with the Drone CI service, but @nick.thomas might have a better idea about what could cause this behaviour.
https://gitlab.com/gitlab-org/gitlab-ce/issues/44046 would help here but I think this issue is slightly different.
cc @erushton