Draft: Consider the Creation of a Data Validation Service
Context
Currently, the registry assumes that data written to common storage is valid at the time of writing and will remain valid indefinitely. Additionally, with database backed metadata, we assume that only the registry will modify the underlying data on the storage system.
We've seen several issues that prove that these assumptions are not correct:
- Investigate corrupted manifest payloads (#333 - closed) - missing data that was valid at time of writing
- Failed Manifest Pushes Due to Layer Linking Err... (#687 - closed) - data invalid at time of writing
- Empty Tag returned by 'v2/<REPO>/tags/list' whe... (#341 - closed) - data modified by external process
Problems
In general, we want any manifest push by tag that succeeds to be pullable indefinitely. The scenarios below enumerate events which cause this not to be true.
User Action
These following events are expected, and the registry handles these appropriately:
- All tags (or other manifests) referencing the manifest are deleted
- The manifest itself is deleted directly
- A layer is unlinked from the manifest via the Blob Delete API
Unlinking the layer leave the manifest in an odd state as it remains visible, but pull will fail with an error. Unfortunately, this is the legacy behavior, and the spec for this API endpoint does not mention manifests which may reference this blob.
Manifest Stored Incorrectly
Manifest Data Changed After Write
While most of the errors we've seen due to data corruption or changes that are made to the underlying storage system via direct user activity have only effected the metadata side of the registry, and will therefore not affect registries running with database metadata and without metadata mirroring turned on.
Blobs Present in Database but Not in Object Storage
Blobs Present in Object Storage, but Not in Database
Blob Data Corruption
Solution
A background process using similar techniques as the online garbage collector could be used to regularly validate registry data.
Manifest Validation
Blob Validation
Blobs Present in Database but Not in Object Storage
Blob Data Corruption
Blobs Present in Object Storage, but Not in Database
This would require enumerating blobs via the object storage driver for a full list of blobs, then checking for the presence of each of blob on the database, adding metadata for the blob if it is not found.
This process is the same as in place imports done by the CLI migration tool with the --dangling-blobs option. This scenario is most likely due to a one time error made during the course of an offline in place migration to the database and does not interfere with registry operations. I believe this is best covered by adding in a flag to the import tool which only inventories the blobs and adds missing ones to the database.
Corrective Action
While it's relatively simple to detect invalid data, taking action on that data is more complex. Since we can anticipate that data corruption is relatively rare, I believe the proper first iteration will revolve around visibility, rather than attempting automated repair.
Once we see how often invalid data is, confirmed a zero rate of false positives, and have developed appropriate manual interventions, we then should consider automated data repair.
Reporting
There are several channels over which invalid data could be reported.