### Problem to solve Migration an internal organization from GitLab.com legacy cell to another cell. ### Intended users * [Sidney (Systems Administrator)](https://handbook.gitlab.com/handbook/product/personas/#sidney-systems-administrator) * [Ingrid (Infrastructure Operator)](https://handbook.gitlab.com/handbook/product/personas/#ingrid-infrastructure-operator) ### User experience goal The infrastructure operator / systems administrator should be able to select an organization to migrate and a target cell to migrate to. This will be a CLI-driven operation initially. It must be possible to track and display the progress of replication and verification at a point during the migration. The background replication should be transparent to the end user / customer. A maintenance window is acceptable to finalize replication of all changes, and to cut over to the target cells. For the first move of an internal organization, if it is somewhat large (i.e. `gitlab-org` + `gitlab-com`), then we should target less than 24 hours of organization downtime. Small organizations should target less than 1 hour of maintenance window for now. ### Proposal All the organization's GitLab data must be copied from GitLab.com to the cell. This includes meta data stored in the rails DB, repositories and files. The copied data must be verified to ensure it has not been corrupted in transit or at rest. Since large volumes of data will be involved, a one-shot approach will not be practical since it will require a large downtime to copy all the data across. Therefore, a continuous replication approach is desired. It must be possible to track the replication progress for an organization throughout the replication process. This can be either through a CLI command or Prometheus metrics upon which a dashboard can be built. Any errors in the replication process are output to a log for further analysis and troubleshooting. The replication process must be tunable to allow the load on the respective systems and the network to be managed so at to not create a degradation in user experience for users on GitLab.com and any users of organizations already on the target cell. Load on these systems from user activities can be expected to fluctuate. It must be possible to dampen the impact from the migration if the systems involved run out of headroom. It must be possible to configure/de-configure the replication on live systems without requiring downtime. Both GitLab.com and the target will have live organizations with users consuming services. We therefore cannot afford to incur downtime for configuring the replication. While replication lag is not a significant concern in this workflow, it should be contained to less than 2 minutes to ensure an efficient migration process and to support a short maintenance window when cutting over services to the cell. A large replication lag will either reduce the time for checks during the maintenance window or require extending the maintenance window. Both are undesirable. This is the first step towards https://gitlab.com/groups/gitlab-org/-/epics/17308+. ### Implementation details After investigating PostgreSQL logical replication to scope the replicated PostgreSQL data to an organization, it was discovered that logical replication would risk saturating the WAL disk I/O throughput for a busy database such as the one for GitLab.com. Therefore, we have decided to take an alternative approach for the MVC. The org mover MVC will be implemented to talk directly to the primary database on GitLab.com. It will directly use this information (instead of relying on read replica like in the case of Geo) to initially replicate the repository and object storage data to the secondary cell. The repository and file data will be replicated continuously in the background during the backfill process. The backfill process is where the secondary cells attempts to catch up with the primary. Once the backfill is complete, changes committed to the organization such as repository commits, file uploads, etc ... will trigger replication events the secondary cells to receive these changes. At a pre-planned time with the internal customer, the organization will be put into maintenance mode where new writes will not be accepted. The repository and file replication for new changes will be allowed to complete. We will then take a backup of the organization's PostgreSQL data on the GitLab.com monolith and copy and restore this on the secondary Cell. Once the PostgreSQL data for the organization has been restored to the secondary Cell, routing logic will be reconfigured to direct requests for the migrated organization to the secondary Cell. At this point, maintenance mode will disabled and the organization will be active on the secondary Cell. ### Rollout plan Several preparatory steps will be undertaken to lay the groundwork for org mover. We will rollout the following changes first on staging and then on production. **1. Enabling checksumming** We will use checksums to verify data has been copied from the GitLab.com monolith to the secondary Cell correctly. I.e. that it has not been corrupted in transit or at rest. The checksumming process will also reveal any orphaned database records and files. The checksumming process will likely put some load on the database. We will closely monitor and tune the concurrency settings. Checksumming will be enabled one [data type](https://docs.gitlab.com/ee/administration/geo/replication/datatypes.html#replicated-data-types) at a time using feature flags to control the rollout. **2. Clean up orphaned records and files** Clean up the orphaned records and files identified when checksumming was enabled. ### Permissions and Security Configuring and initiating the replication will require administrator access to GitLab along with CLI access. ### Documentation ### Availability & Testing ### Available Tier * Premium/Silver * Ultimate/Gold ### Feature Usage Metrics TBD ### What does success look like, and how can we measure that? We successfully moved an internal organization from the current GitLab.com legacy cell to another cell. ### Is this a cross-stage feature? Potentially yes. ### What is the competitive advantage or differentiation for this feature? Org mover will eventually allow organizations to be moved to our scalable cells architecture, future-proofing our GitLab.com services as GitLab grows. Guaranteeing we are able to deliver a consistently great service for our GitLab.com customers with all the advantages that come with the cells architecture. ### Links / references ### Participants * `@dbalexandre` * `@mkozono` * `@s_murray` * `@nsilva5` <!-- STATUS NOTE START --> ## Status 2025-07-08 :tada: **achievements**: - Working on defining a plan for Protocells with Thong [here](https://docs.google.com/document/d/1OmBC8WfHUo6_esSMIeiPun0OJ_tkB3MwVzBJ3katDqw/edit?tab=t.0) (July 2nd Org Mover Sync Agenda) and opening epics/issues based on concrete action items to add to the Protocells [Epic](http://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1626) :issue-blocked: **blockers**: - Mike is also the customer support DRI this week so it's been hard to get enough time to work on the roadmap items. Other team members are going to fill in this role so we can free up Mike's time to get interlock/roadmap/epics done for Protocells. :arrow_forward: **next**: * Finishing the high level epic planning for Protocells and adding epics and issues to the Protocells [Epic](http://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1626) _Copied from https://gitlab.com/groups/gitlab-org/-/epics/12859#note_2610441962_ <!-- STATUS NOTE END -->