Add org-mover manifest endpoint to the internal API
What does this MR do and why?
Adds GET /api/v4/internal/org_mover/manifest. It returns this cell's Org Mover manifest (orgmover.manifest.v1.Manifest as JSON): the databases, tables, object storage sources and Gitaly sources that an organization move must carry. GitLab generates it on each request from the table dictionary (db/docs/*.yml) and the live columns, so it matches the running GitLab version.
Today, Org Mover gets the manifest from a script, demo/cng/scripts/manifest.rb, that an operator runs by hand for each GitLab version. When the schema changes and nobody runs the script again, a move can leave data behind without a warning. With this endpoint, Rails is the source of truth. This is the Rails side of Get the manifest from the running GitLab (transport#126).
This MR continues Raynard's draft (!266844). It keeps his 3 commits and their authorship, rebased onto master. It adds the fixes that the draft needed to pass CI, and fixes for review findings:
- Fix the manifest spec. It looked for an
uploadssource, butuploadsis the partitioned parent table. It is ingitlab_main_cell_localand has no sharding key, so the manifest never lists it. The spec now checksproject_uploads. - Fix the RuboCop offenses. The feature flag check now uses the
:instanceactor. - Document the endpoint in
doc/development/internal_api/org_mover.md. - Regenerate
config/routing/gitlab_routes.json, the fine-grained token REST docs andopenapi_v3.yaml. - Remove two handler map entries that matched no moved table (
ci_job_artifacts, dropped in 18.1, and the cell-localuploadsparent), and add a spec that fails when a mapped table is not in the manifest. A mapped table that drops out of the manifest would otherwise disappear without an error. - Build the manifest once per spec file instead of once per example, and add specs with a fixed dictionary for the filter branches that the live
db/docsnever takes. - Read the columns of all tables with one
pg_attributequery per connection, instead oftable_exists?andcolumnsfor each table. A request ran about 2,000 catalog queries and took about 11 s on Caproni, more than Org Mover's 10 s fetch timeout. Now it runs 3 queries in 0.26 s. This also keeps the[]of array column types, whichColumn#sql_typedrops, so the output now differs from the script's for 111 array columns.
Parts of the change:
lib/gitlab/org_mover/manifest.rb: the generator, a port of the script.lib/api/internal/org_mover.rb: the route. It uses the same ES256 JWT authentication as the other org_mover routes.org_mover_manifest_apifeature flag (wip, off by default). The manifest describes the cell, not an organization, so it has its own instance-wide flag instead of the per-organizationorg_mover_maintenance_apiflag.
Not in this MR:
- The handler maps (
OBJECT_STORAGE,GITALY) stay curated in Rails, the same as in the script. Moving them intodb/docs/*.ymland deleting the copy in org-mover is the end state of transport#126, and is a follow-up. - Design questions that the port inherits from the script: bot rows that an organization owns, and the script's
not-movedrecord. A GitLab version field in the manifest needs a proto field in Org Mover first.
Follow-up: Some gaps in the manifest that GitLab serves need a decision (transport#601).
The new route is in the HTTP Router snapshot through the paired HTTP Router MR (http-router!1511), which is merged. The route needs no routing change: like the other org_mover routes, Org Mover calls it on each cell's own GitLab service, not through the HTTP Router.
References
Related to gitlab-com/gl-infra/tenant-scale/transport/team#536
- transport#126: Get the manifest from the running GitLab
- Original draft (!266844)
- Paired HTTP Router MR, merged (http-router!1511)
- Follow-up (transport#601)
- Org Mover manifest script
Screenshots or screen recordings
Not applicable: no UI change.
How to set up and validate locally
Evidence, from GitLab in Caproni edit mode, at this branch's head:
spec/lib/gitlab/org_mover/manifest_spec.rbandspec/requests/api/internal/org_mover_spec.rb: 69 examples, 0 failures. With SimpleCov,scripts/undercoverageagainst the merge base reports no missing coverage; the fixed-dictionary specs cover the branches the livedb/docsnever takes. Withci_job_artifactsput back in the handler map, the new map spec fails withexpected ["ci_job_artifacts"].empty? to be truthy. With the per-tabletable_exists?andcolumnslookup put back, the query-budget spec and the array-type spec fail. Each of these also fails one spec:database_foralways returninggitlab_main, the query withoutpg_table_is_visible, and the query without the type modifier. Before the spec fix, theuploadsexample failed withexpected nil to include {handler: "OBJECT_STORAGE_HANDLER_UPLOADS", ...}, the same as in the draft's last pipeline.- Output matches the script: I ran the org-mover script with
rails runner - < manifest.rband serializedGitlab::OrgMover::Manifest.new.generatethe same way. Before the single-query change, withoutgenerated_at, the two JSON documents were byte-identical (999 tables, 37 object storage sources, 5 Gitaly sources). After it, the output is identical to the old output once the[]suffix is removed from array types.generatewent from 2,016 queries and 11.64 s to 3 queries and 0.26 s. The compact response is 556,665 bytes. - RuboCop on the changed Ruby files: no offenses.
rake gitlab:cells:routes:updated_check,rake gitlab:permissions:validateandrake gitlab:openapi:v3:check_docspass.markdownlint-cli2and Vale on the docs page: no errors or warnings.
To try the endpoint, enable the flag and configure an ES256 key pair as Gitlab::OrgMover::Authenticator describes:
curl --header "Gitlab-Org-Mover-Api-Request: <jwt>" "https://gitlab.example.com/api/v4/internal/org_mover/manifest"MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Generated by Claude Opus 5.5 in Claude Code (T3 Code)
Session baef61c8-20b4-40f1-9267-f27e9e05f8ae