Geo: SSF verification retry bugs
Problems
staging.gitlab.com has package file checksum failures which surfaced the following problems with failure retry logic:
- When an error is raised in
track_checksum_result!
while attempting to transition toverification_succeeded
, some fields have already been cleared in-memory during thebefore_transition
block. This causes that kind of failure to always setverification_retry_count
to 1, instead of incrementing it. -
verification_failed_batch_relation
andneeds_verification_relation
are not excluding things which have averification_retry_at
in the future
Proposal
-
Call reset
before callingverification_failed_with_message!
in the rescue https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/gitlab/geo/verification_state.rb#L222? -
Scope verification_failed_batch_relation
andneeds_verification_relation
towhere("verification_retry_at < NOW()")
-
Adjust index package_file_registry_needs_verification
if needed -
Adjust index package_file_registry_failed_verification
if needed
Edited by Michael Kozono