ReactiveCaching concern (online terminal, external CI services, etc) updates its values too aggressively

Summary

As noted by @DouweM .

Various parts of the GitLab codebase need to call out to external HTTP services. We don't want this to happen within the unicorn request/response lifecycle so we wrap these calls in the reactive cache, which executes them in sidekiq and writes the value to redis.

A lifetime and a refresh interval is defined for these cache values. We're only meant to update the value once per refresh interval, but in face, we attempt to update it every time the value is read.

Steps to reproduce

Set up an external CI service (e.g., Jenkins, TeamCity) or the online terminal

Spam the "commit status" or "terminals" endpoint, depending on which you set up

What is the current bug behavior?

The rate of ReactiveCachingWorker jobs enqueued, and the number of HTTP requests made to the external service, is much higher than 1/refresh interval.

What is the expected correct behavior?

The rate of ReactiveCachingWorker jobs enqueued, and the number of HTTP requests made to the external service, should be ~1/refresh interval.

Possible fixes

Stop enqueuing the worker so frequently