Request for Additional Resourcing for Protocells - Org Mover Version
Summary
We are requesting additional resourcing that we have identified will accelerate/derisk the delivery plan for Protocells - Org Mover. We are identifying two different ways we can do this by getting different set of expertise. 1 way is to get Siphon/PG help and another is SRE support. In the following sections we will present details of both approaches and explain the reasoning behind each.
Siphon/PG
Start Date - ASAP
How many days per week - full time until Siphon for Org Mover is implemented so we can have this in time for Protocells.
What they would work on - Implementing/accelerating Org Mover with Siphon/Logical Replication, this was originally started on by Adam/Mike and still needs to be fleshed out further and a lot of implementation needs to be done and issues creates/broken down. Without this assistance, the process for migrating PostgreSQL data will be much more primitive and create long downtime, see gitlab-org&18462.
Relevant Documentation/Links
- Org Mover: Logical replication with row filters (gitlab-org/gitlab#523306) - LR with row filters
- Explore leveraging Siphon for Org Mover (gitlab-org/gitlab#525187 - closed) - Original POC issue which Adam took an attempt at adding a consumer, testing still needs to follow.
SRE Support
Start Date - By end of July 2025 once we identify all the roadmap items and know how to tackle a plan
How many days per week - initially consulting only and then full time as we start testing on staging/production
What they would work on - Since the migration to Protocells begins in FY27Q1, less of the process will be built into the product. An SRE with a runbook and Ansible playbooks will be driving the migrations. Game days moving test organizations should start at least 2 months prior, and should be driven by the SREs who will be doing it for real. Development should be dictated by the needs of these SREs. Much of the development will not be in Rails, but in scripts and Ansible playbooks, so it is better for an SRE to coordinate the work of the Rails engineers.
Relevant Documentation/Links
- The overall plan is Organization Data Migration (gitlab-com/gl-infra&1626)
- An SRE is the most appropriate role to lead the development of all subepics except for one (Organization migration: Checksum/sync by org on... (gitlab-org&14631))
- For reference, a sequence diagram of migrating an organization
Backend Engineer Support (with Rails Experience and Preferable Geo Experience)
Start Date - ASAP
How many days per week - full time until Protocells/Organizational Data Migration is implemented so we can have this in time for Protocells.
What they would work on - Rails implementation of Organization Data Migration, we lack enough capacity on the team with the current engineers to carry this work out in time for October so we can be ready for Protocells in January. Please see #2 (comment 2644793258) for more details.
Relevant Documentation/Links
- Organization migration: Checksum/sync by org on... (gitlab-org&14631) - Epic for all the rails work