### Problem to solve Common data integrity problems with package files: * The file doesn't exist * The file changed (checksum mismatch) * The associated model does not exist (i.e. a project record was deleted) * Geo only: The file doesn't exist on a secondary * Geo only: The file changed on a secondary These problems can arise due to bugs or transient infrastructure problems. ### Use case: Geo DR Sysadmins need *assurance* of data integrity or at least an indication that data needs to be fixed, in order to trust a DR solution. ### Proposal * Add service to verify package files on secondaries * Add service to checksum package files on primary * Automatically verify package files on secondaries * Ensure package file checksum on primary * Track primary package file checksum counts * Track secondary package file verification counts * When a primary package file is checksummed, reset verification on secondaries * Regularly reverify package files on primary * Add package files checksum and verification progress bars to Geo Nodes UI * Let admins adjust file verification rate * Allow verification of files in Object Storage ### What does success look like, and how can we measure that? ### Testing Please refer to [the testing section of the parent epic](https://gitlab.com/groups/gitlab-org/-/epics/2161#testing) #### Available Prometheus metrics * Primary: Number of checksummed files * Primary: Number of failed checksummed files * On secondaries: Number of verified files * On secondaries: Number of failed verification files #### In `Admin Area > Geo > Nodes` Each node should display Upload a checksum/verification progress bar using the above data (in GeoNodeStatus). The [current limitations table](https://docs.gitlab.com/ee/administration/geo/replication/#limitations-on-replicationverification) should look like this: | Data Type | Replicated | Verified | |-----------|------------|----------| | Package files | Yes | Yes |