Horizontal Pod Autoscaler for sidekiq considered harmful
The title is hyperbole (couldn't resist; sorry/not sorry), but carries a grain of truth.
We have multiple cases where the auto-scaler of pods in kubernetes is problematic for sidekiq. While there are a large number of jobs that can and do complete quickly (<10 seconds), there are a small handful which are long and slow. The obvious cases are imports + exports, and we're dealing with imports in gitlab-org/gitlab#332616 (closed).
Another less obvious case is ArchiveTraceWorker and ArchiveTracesCronWorker, which has been a problem for months now (see gitlab-org/gitlab#330141 (closed)) but which we (well, I) only just realized is potentially (in large part) the same situation, where sometimes these jobs need to take up to 10s of minutes (needs to read then delete 10K chunks in object storage, at 15 ops/s), and the HPA is so aggressive ambitious that the jobs either never, or rarely, get a chance to run that long.
There may be others (I will look and update; any real problem children will show up with high interrupted_count and in perhaps in the interrupted set).
Proposal: A "long running" or "stable" shard for these sorts of jobs, with (as for the recently new import shard) no HPA (same min/max replicas), running jobs identified as needing that sort of stability. We can either use a new tag, or perhaps add a new resource_boundary (runtime=long
perhaps?) I think I like the resource_boundary better than a tag for aesthetics/consistency, but I'd take anything to solve this issue.
@skarbek Thoughts? Thinking of you as you're knee-deep in the new imports shard right now. Maybe we rename/rejig?
@smcgivern Thoughts on resource boundaries vs tags (or other options for identifying the relevant jobs)?