Skip to content

Geo: Increase verification backfill job exclusive lease timeout

Michael Kozono requested to merge mk/bump-verification-state-backfill-worker-lease-timeout into master

What does this MR do and why?

Describe in detail what your merge request does and why.

I know we need to increase it because I've observed the job take longer than 1 minute in staging.gitlab.com. As an upper bound, I don't have specific data showing 30 minutes to be ideal, but I'd be very surprised if it took longer than 30 minutes in even a slow environment. And it's ok if it's too long, since the lease is released upon completion of the job. If the job dies without releasing the lease, then this background task will take up to 30 minutes to be attempted again (it's triggered by a cronjob each minute), which is acceptable.

More background: The exclusive lease is used to make sure only one of these jobs is running at a time, since it's somewhat resource intensive but not urgent to complete in a specific amount of time. If the lease timeout is too short, then multiple of these jobs will run at a time, which is unnecessarily wasteful.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Michael Kozono

Merge request reports