Parallelize artifact deletion
Have you thought of breaking the list of files into batches and run a worker for each batch? From a Redis perspective the usage is pretty much the same but we can (1) parallelize the deletion and (2) having a shorter list it would ensure Sidekiq job doesn’t timeout and restart.
artifact_files.each_slice(DELETE_BATCH_SIZE) do |files| Ci::DeleteStoredArtifactsWorker.perform_async(files) end