Introduce idempotent worker attribute
As discussed at &96 (comment 228932991), Sidekiq encourages making every worker idempotent, i.e. meaning it should be able to run multiple times without side-effects, though we still experience issues in that regard (gitlab-org/gitlab#32737 (closed), gitlab-org/gitlab#18500 (closed)).
There are a couple of things we might do to improve the reliability on that such as adding integration tests for each worker (&96 (comment 229002255)) ensuring that multiple calls (on even half-processed jobs) are still idempotent. Though it's not a catchall solution (given jobs can fail in multiple places, for multiple reasons, leaving unpredictable state behind - it alone can be tricky to ensure with a few tests), but it's a starting point.
gitlab-org/gitlab!25202 (merged) does:
- Introduce the
idempotent!
method for workers - Add a cop to catch workers that don't call it in the scope of the job class
- Set
ExpireJobCacheWorker
as idempotent, adding tests with introduced helper methods / shared example