Geo: Build a scalable, self-service geo replication and verification framework
> Hand some data to an engineer and they will not replicate it. Teach an engineer how to replicate data and customer happiness shall be with you. *- The Wise Geologist*
### Introduction
There are [several discussions](https://gitlab.com/gitlab-org/geo-team/discussions/issues/45) within the Geo team regarding the limitations of our current replication system. As of October 2019 [only ~50% of data types](https://docs.gitlab.com/ee/administration/geo/replication/#current-limitations) (we need a better name) are replicated and of those only ~41% are fully verified. This is known and we have made some efforts to change this situation by trying [to replicate the remaining data types](https://gitlab.com/groups/gitlab-org/-/epics/893) and [by trying to verify those data types](https://gitlab.com/groups/gitlab-org/-/epics/1430). As part of those efforts we learned that replicating data types is hard and so is verifying those data.
There are several technical reasons for this, including the current architecture, the differences in data types, usage of FDW in combination with selective sync etc.; however, these are not solely responsible for the difficulties. GitLab is growing rapidly and teams across the organisation are adding features at a rapid pace. Many of these features add new data types and are not initially designed to be easy to replicate; for example, GitLab Pages, server-side hooks, and Design Repositories. This is very likely through a lack of knowledge and because Geo is not considered during initial designs. Engineers across the company are not empowered to easily support geo replication and consequently the Geo team becomes a bottle neck.
In order to address both the technical challenges and the operational limitations, I propose to build a new geo replication and verification framework with the explicit goal of enabling teams across GitLab to add new data types in a way that supports geo replication out of the box. It should be incredibly easy for engineers to do the right thing. The Geo team should develop the framework and offer support to the organisation but would no longer be responsible for most of the implementation.
### Problem to solve
* Geo is usually not considered by other teams when implementing features
* Customers expect new features (and their data) to be replicated either for performance or Disaster Recovery purposes
* Adding new data types to Geo is hard and can only be performed by the Geo team
* Verification of data types is difficult and does not perform well. Again only the Geo team can do this
* The company is growing rapidly and adds new features; the current operational mode is not scalable
* Software developers across GitLab are not empowered to make their features geo-compatbile
<!-- What problem do we solve? -->
### Intended users
* [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer)
### Further details
There are currently some technical considerations on how to iterate on Geo:
* https://gitlab.com/gitlab-org/gitlab/issues/33624
* https://gitlab.com/gitlab-org/gitlab/issues/12565
* https://gitlab.com/gitlab-org/geo-team/discussions/issues/45
<!-- Include use cases, benefits, and/or goals (contributes to our vision?) -->
### Proposal
<!-- How are we going to solve the problem? Try to include the user journey! https://about.gitlab.com/handbook/journeys/#user-journey -->
* Create a Geo replication framework that is so easy to use that every software developer in GitLab can utilise it to make a new feature "Geo compatible"
* The framework should abstract away many of the low level functionalities e.g. verification so engineers don't need to worry about them
* Create educational material, workshops etc. to teach folks how to use it.
@toon wrote up some pseudocode of how this could look like:
```ruby
class MyCoolNewFeatureModel < ApplicationRecord
include Geo::Replicable
geo_replicate_repository :cool_repository # name/prefix of the column(s) where Geo can find the repo
# rest of the code unrelated to Geo
# ...
end
```
### Documentation
<!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html
Add all known Documentation Requirements here, per https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements
If this feature requires changing permissions, this document https://docs.gitlab.com/ee/user/permissions.html must be updated accordingly. -->
* We would need to create documentation for this. The better the documentation, the more likely this is to take off
### Testing
<!-- What risks does this change pose? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? See the test engineering process for further help: https://about.gitlab.com/handbook/engineering/quality/test-engineering/ -->
* Geo replication must be extensively tested on all levels because of its relevance for Disaster Recovery
##### End-to-end tests
Each Geo replicable should have its own end-to-end test with the same basic structure:
- Create/upload Geo replicable to primary site (may require adding test fixtures for each new Geo replicable)
- Confirm replication on secondary site
- Possibly independently verify checksums on primary and secondary (check other testing levels for coverage on this)
- Confirm verification status of specific Geo replicable (for example on /admin/geo/{geo_replicable} page on secondary site)
Ideally we can leverage existing test code for these Geo replicables, and focus on the replication process.
##### Test documentation
Currently we don’t provide end-to-end test templates in our QA test framework. In addition, the page objects and navigation used in each test will be specific to each data type. Instead of creating a test template file we could add to existing end-to-end testing docs (https://docs.gitlab.com/ee/development/testing_guide/end_to_end/) and link it to the self-service framework documentation.
### What does success look like, and how can we measure that?
<!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. -->
* Percentage of new features that are geo replication and DR ready out of the box (target: 80% of new features implemented by GitLab support Geo replication and DR out of the box)
* Time it takes for software developer to become productive using the Geo framework
* Number of steps needed to make a new data type geo-compatible
### What is the type of buyer?
<!-- Which leads to: in which enterprise tier should this feature go? See https://about.gitlab.com/handbook/product/pricing/#four-tiers -->
* Premium
* Ultimate
### Links / references
epic