WIP: Verify repositories on the Geo secondary

What does this MR do?

As outlined in #4754 (closed) and #4469 (closed), we want to verify repositories on the secondary match the repositories on the primary.

Assumption: a project's checksum is stored with the project in the database, and gets replicated into the secondary.

• Add new verification/checksum columns in the project_registry table of the Geo tracking database
• Each time a project is synced on the secondary, it's checksum is cleared in the ProjectRegistry.
• Every 6 hours, a job kicks off that scans the ProjectRegistry looking for projects that have not been checksumed.
• If it's stable (project has not been synced in 6 hours and the project has a recently computed checksum), then compute the checksum and verify it matches the main project checksum.
• If the checksum fails, mark it as failed in the ProjectRegistry.
• Enhance the API GET /geo_nodes/current/failures to return the failures.
• Show status of failures and stats to admin user. Consider adding a "Data Integrity" or "Repository Verification" panel to the "Monitoring" section
• Use multiple Sidekiq jobs to verify all of GitLab.com, careful not to cause an inordinate amount of CPU and I/O load

Are there points in the code the reviewer needs to double check?

You can take a look at Geo::RepositoryVerificationWorker#should_verify_repository?(registry, type) to double check the conditions for determining that a verification is needed.

Why was this MR needed?

To detect any integrity problems with synced repositories

Does this MR meet the acceptance criteria?

• Changelog entry added, if necessary
• Documentation created/updated
• API support added
• Tests added for this feature/bug
• Review
  • Has been reviewed by UX
  • Has been reviewed by Frontend
  • Has been reviewed by Backend
  • Has been reviewed by Database
• Conform by the merge request performance guides
• Conform by the style guides
• Squashed related commits together
• [-] Internationalization required/considered
• [-] If paid feature, have we considered GitLab.com plan and how it works for groups and is there a design for promoting it to users who aren't on the correct plan
• End-to-end tests pass (package-qa manual pipeline job)

What are the relevant issue numbers?

Closes #4754 (closed)