`PipelineScheduleWorker` and `ExpireBuildArtifactsWorker` are being killed by Sidekiq Memory killer
It seems the cron workers
ExpireBuildArtifactsWorker eat up a few GB of RAM, and because of that, Sidekiq memory killer keeps killing the processes.
You can see the datapoint from the following Grafana link.
This is actually worse than it looks. To illustrate the problem, I describe how the system behaves below.
PipelineScheduleWorkerfires according to the cron schedule
19 * * * *. This is an hourly worker.
PipelineScheduleWorkercreates an exclusive lock with Redis to prevent concurrent running.
PipelineScheduleWorkereats up RAM over 1GB. Due to our current RSS limit is 1GB, the process gets
SIGTSTP, and eventually
PipelineScheduleWorkerfires again (regardless of its
sidekiq_options retry: false), however, this process fails immediately, because the exclusive lock created at the 2. still remains.
- In nutshell,
PipelineScheduleWorkerwill never complete the job.
- Reduce the memory consumption on the worker
- Remove exclusive lease
PipelineScheduleWorkerresilient (currently, multiple pipelines can be created in a short interval because of