Add docs for idempotent workers

What does this MR do?

It adds docs on how to mark jobs as idempotent and adding tests for it.

Main MR: !25202 (merged).

Closes gitlab-com/gl-infra/scalability#179 (closed)