Geo: Improve Geo scalability by simplifying backfill operations
Engineering Owner: @dbalexandre ### Problem to solve <!-- What problem do we solve? --> GitLab Geo uses PostgreSQL [Foreign Data Wrappers (FDW)](https://wiki.postgresql.org/wiki/Foreign_data_wrappers) to perform some cross-database queries between the primary and secondary Geo nodes. Despite the technical elegance of this approach, these queries have lead to a number of problems for Geo. These include increased complexity, which makes upgrading harder, scalability issues on older postgres versions (9.6), and maintainability issues. Due to some initial work from @mkozono in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20002 we know that moving Geo to a more event-based system will eliminate most of these concerns. It follows, that removing FDW will solve most of these issues. ### Intended users This is relevant for all Geo users. ### Further details <!-- Include use cases, benefits, and/or goals (contributes to our vision?) --> Initial architecture for LFS files: ![LFS_backfill](/uploads/106cb0289d08087c332416cccd1be2a8/LFS_backfill.png) ### Proposal * Make the Registry table the single source of truth * Implement for each relevant data type * Eliminate any other occurence of FDW * Delete a bunch of documentation * Rejoice ### Permissions and Security ### Documentation This is mainly a backend change and will require mostly Developer documentation. We will be able to remove some documentation that is specific to FDW ### Testing <!-- What risks does this change pose? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? See the test engineering process for further guidelines: https://about.gitlab.com/handbook/engineering/quality/guidelines/test-engineering/ --> This is a major architectural change and all changes should be tested in staging at scale before removing feature flags. ### What does success look like, and how can we measure that? * All datatypes can be replicated without FDW * FDW documentation eliminated ### What is the type of buyer? <!-- Which leads to: in which enterprise tier should this feature go? See https://about.gitlab.com/handbook/product/pricing/#four-tiers --> * Ultimate * Premium ### Links / references * https://wiki.postgresql.org/wiki/Foreign_data_wrappers
epic