Add periodic background verification for repository metadata
Problem
Praefect lacks background verification of the replicas and their associated metadata. Incorrect metadata records at the moment can mostly cause Praefect to believe a replica exists where none exists. This can lead to errors when routing requests to non-existent replicas. It can also lead to data loss that is not exposed as Praefect reports the data loss purely on the metadata records. The metadata can get out of sync if changes happen on the disks without going through Praefect, for example when manually modifying repositories or due to disk failures. There may also be invalid records due to bugs in the state management logic.
Solution
As a first iteration towards verifying the data stored in the cluster, we should add a background verifier in Praefect that verifies whether the replicas indicated by the metadata actually exist.
This is a good first step as following #3485 (closed), the repository creations and deletions are atomic and the repositories are not moved on the disk. Given that, we should never encounter races that could result in incorrect decisions. If a metadata record exists, the repository should exist on the disk. It's possible that the repository is deleted while the verification is in process. In such case, the metadata record is still deleted first so the verifier can simply ignore such errors.
Invalid metadata records should be removed and the removal logged.
Implementation
As the replicas need to be verified periodically, we need to store a timestamp indicating when the replica was last verified. The timestamp allows for:
- Ensuring fairness in that every replica gets verified eventually while prioritizing the ones have had the longest time since the last verification.
- The verification process should work in batches and acknowledge successes in between. Updating the timestamp in between allows the work to be continued from where it left after process restarts.
- Coordinates between the various Praefect processes so they don't verify the same replicas multiple times.
In addition, we need a replica specific lease that is acquired by a Praefect about to verify the replica. This prevents multiple Praefects from verifying the same replica at the same time. Lease says until what point no other Praefect should pick up verification for this repository. Since verification is not that time critical, we can make the lease something reasonably long like 30 seconds. That way we don't need background updating of locks similar to what we use in the replication queue.
These can be added into the storage_repositories
table as verified_at timestamp
and verification_leased_until timestamp
.
These can then be indexed to quickly determine repos that need verification, for example.
CREATE INDEX verification_queue ON storage_repositories ( verified_at NULLS FIRST )
WHERE verification_leased_until IS NULL;
The verification workers would then query for replicas with earliest verified_at
prioritizing records with verified_at
being null. For each of the replicas, the worker calls RepositoryExists
. Once the results are in, the worker updates the verified_at
and releases the lease by setting verification_leased_until = null
.
To release stale leases, there should be a goroutine that periodically set all stale leases to NULL. Alternative would be to include the verification_leased_until
in the verification_queue
index and picking up replicas which have a stale lease, but that would lead to the workers picking up work doing more work on skipping already acquired verification jobs. To identify stale leases fast, another index would be added
CREATE INDEX stale_verification_leases ON storage_repositories ( verification_leased_until )
WHERE verification_leased_until IS NOT NULL
The stale lease releasing background goroutine then queries the leases that have been exceeded and sets them to null.
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.