> *Data integrity is essential to Geo. If the data is worth replicating, then it's worth verifying.* ~ @mkozono Engineering Owner: @brodock ### Summary (for the Release Post; describes potential future state) Geo replicates data from a **primary** node to one ore more **secondary** nodes. We now ensure that all data that is transferred from a **primary** to the **secondaries** is automatically verified after it was transferred, so users can be sure that no data was corrupted during the transfer. This is especially important in a disaster recovery scenario. We also surface information in the Geo administrator dashboard to allow systems administrators to monitor events and re-sync data if needed. ### Problem to solve <!-- What problem are we solving for them? Tight problem description that everyone can rally around. --> Geo replicates data from a **primary** node to one ore more **secondary** nodes. This replication happens via different mechanisms but it will result in different copies of the data being stored on different nodes. During the transfer, files may get corrupted and this can cause issues when reading files from the **secondary** or when a **secondary** is promoted to a **primary** as part of a failover. In order to solve this problem, checksums, such as SHA256, can be calculated before the transfer and compared to the checksum of the copied file after the transfer. If the checksums are identical, the file was transferred successfully. We need to implement a system for Geo that performs these checks, ideally automatically, and then takes corrective action. ### Proposal * Create a list of all data that are available in GitLab on the primary, which ones are replicated, and which ones are automatically verified. This is being addressed in [Compile a list of created data types, which ones are replicated, verified and tested](https://gitlab.com/gitlab-org/gitlab-ee/issues/12398) * Add verification mechanisms for all replicated data sources based on an [appropriate hash function](https://en.wikipedia.org/wiki/SHA-2). This may be manually triggered, for example via the UI. We will tackle the automation in a later step. * Automate the verification process after the manual implementation * Take corrective action when transferred data is detected (e.g. re-try the transfer) * Surface the information to users e.g. when a large number of transfers fail the connection between two nodes may be unstable. * Make data verification a part of the ongoing development effort. Every time a new data source is added for replication, transfers must be verified. ### Higher intent Many customers use Geo as a DR solution. Verifying replicated data is crucial in a DR situation because otherwise users can't trust that the replicated data is actually correct. If Geo is used as a DR solution trust in the the data is crucial. This will be a step to increase the trust in the system. ### Intended users <!-- Who's the target user? Target user description. --> * [Systems administrators](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sidney-systems-administrator) * [Software developers](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer) ### Further details ### What does success look like, and how can we measure that? - Success is when all replicated data sources are verified automatically after replication, corrective action is taken automatically if possible and information is surfaced in the UI ### What is the type of buyer? * Premium ### Links / references /label ~feature