Skip to content

WIP: POC: Ensure worker tests perform multiple times without exception

Oswaldo Ferreira requested to merge osw-ensure-idempotency-for-jobs-part-1 into master

What does this MR do?

That's part of the work to achieve gitlab-com/gl-infra/scalability#42 (closed). In order to mark workers as idempotent (using WorkerAttributes) we first need to make sure:

  1. The worker can perform multiple times without exceptions
  2. The behavior of running it once or multiple times with the same arguments should generate the desired side-effect only once. For instance, a MergeWorker should merge and update the DB in the first run, and running it again should not raise exceptions or change the MR in a unexpected way.

The goal of this MR is ensuring (in a unit-test level) 1 and 2, and we do so by:

  1. Making sure to use the new shared example that runs the job multiple times
  2. Using the new perform_multiple method instead calling worker.perform directly, keeping expectations as they are.

Ideally we should have a cop that ensures the usage of perform_multiple with default arguments (it'll run the job 3 times), instead issuing a worker.perform. If for instance there are stubbed expectations, it's OK to skip the cop and use a perform_multiple(args, exec_times: 1), but a scenario with perform_multiple without these stubbed expectations should exist (so we can ensure the stack is running further down).

It's also OK to use perform_multiple(args, exec_times: 1) (and skip the cop) if it's a mailer-like worker, as we're unlikely to mark these as idempotent.

gitlab-com/gl-infra/scalability#92 (closed)

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Oswaldo Ferreira

Merge request reports