Add org-mover manifest endpoint to the internal API

What does this MR do and why?

Adds GET /api/v4/internal/org_mover/manifest. It returns this cell's Org Mover manifest (orgmover.manifest.v1.Manifest as JSON): the databases, tables, object storage sources and Gitaly sources that an organization move must carry. GitLab generates it on each request from the table dictionary (db/docs/*.yml) and the live columns, so it matches the running GitLab version.

Today, Org Mover gets the manifest from a script, demo/cng/scripts/manifest.rb, that an operator runs by hand for each GitLab version. When the schema changes and nobody runs the script again, a move can leave data behind without a warning. With this endpoint, Rails is the source of truth. This is the Rails side of Get the manifest from the running GitLab (transport#126).

This MR continues Raynard's draft (!266844). It keeps his 3 commits and their authorship, rebased onto master. It adds the fixes that the draft needed to pass CI, and fixes for review findings:

  • Fix the manifest spec. It looked for an uploads source, but uploads is the partitioned parent table. It is in gitlab_main_cell_local and has no sharding key, so the manifest never lists it. The spec now checks project_uploads.
  • Fix the RuboCop offenses. The feature flag check now uses the :instance actor.
  • Document the endpoint in doc/development/internal_api/org_mover.md.
  • Regenerate config/routing/gitlab_routes.json, the fine-grained token REST docs and openapi_v3.yaml.
  • Remove two handler map entries that matched no moved table (ci_job_artifacts, dropped in 18.1, and the cell-local uploads parent), and add a spec that fails when a mapped table is not in the manifest. A mapped table that drops out of the manifest would otherwise disappear without an error.
  • Build the manifest once per spec file instead of once per example, and add specs with a fixed dictionary for the filter branches that the live db/docs never takes.
  • Read the columns of all tables with one pg_attribute query per connection, instead of table_exists? and columns for each table. A request ran about 2,000 catalog queries and took about 11 s on Caproni, more than Org Mover's 10 s fetch timeout. Now it runs 3 queries in 0.26 s. This also keeps the [] of array column types, which Column#sql_type drops, so the output now differs from the script's for 111 array columns.

Parts of the change:

  • lib/gitlab/org_mover/manifest.rb: the generator, a port of the script.
  • lib/api/internal/org_mover.rb: the route. It uses the same ES256 JWT authentication as the other org_mover routes.
  • org_mover_manifest_api feature flag (wip, off by default). The manifest describes the cell, not an organization, so it has its own instance-wide flag instead of the per-organization org_mover_maintenance_api flag.

Not in this MR:

  • The handler maps (OBJECT_STORAGE, GITALY) stay curated in Rails, the same as in the script. Moving them into db/docs/*.yml and deleting the copy in org-mover is the end state of transport#126, and is a follow-up.
  • Design questions that the port inherits from the script: bot rows that an organization owns, and the script's not-moved record. A GitLab version field in the manifest needs a proto field in Org Mover first.

Follow-up: Some gaps in the manifest that GitLab serves need a decision (transport#601).

The new route is in the HTTP Router snapshot through the paired HTTP Router MR (http-router!1511), which is merged. The route needs no routing change: like the other org_mover routes, Org Mover calls it on each cell's own GitLab service, not through the HTTP Router.

References

Related to gitlab-com/gl-infra/tenant-scale/transport/team#536

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

Evidence, from GitLab in Caproni edit mode, at this branch's head:

  1. spec/lib/gitlab/org_mover/manifest_spec.rb and spec/requests/api/internal/org_mover_spec.rb: 69 examples, 0 failures. With SimpleCov, scripts/undercoverage against the merge base reports no missing coverage; the fixed-dictionary specs cover the branches the live db/docs never takes. With ci_job_artifacts put back in the handler map, the new map spec fails with expected ["ci_job_artifacts"].empty? to be truthy. With the per-table table_exists? and columns lookup put back, the query-budget spec and the array-type spec fail. Each of these also fails one spec: database_for always returning gitlab_main, the query without pg_table_is_visible, and the query without the type modifier. Before the spec fix, the uploads example failed with expected nil to include {handler: "OBJECT_STORAGE_HANDLER_UPLOADS", ...}, the same as in the draft's last pipeline.
  2. Output matches the script: I ran the org-mover script with rails runner - < manifest.rb and serialized Gitlab::OrgMover::Manifest.new.generate the same way. Before the single-query change, without generated_at, the two JSON documents were byte-identical (999 tables, 37 object storage sources, 5 Gitaly sources). After it, the output is identical to the old output once the [] suffix is removed from array types. generate went from 2,016 queries and 11.64 s to 3 queries and 0.26 s. The compact response is 556,665 bytes.
  3. RuboCop on the changed Ruby files: no offenses.
  4. rake gitlab:cells:routes:updated_check, rake gitlab:permissions:validate and rake gitlab:openapi:v3:check_docs pass.
  5. markdownlint-cli2 and Vale on the docs page: no errors or warnings.

To try the endpoint, enable the flag and configure an ES256 key pair as Gitlab::OrgMover::Authenticator describes:

curl --header "Gitlab-Org-Mover-Api-Request: <jwt>" "https://gitlab.example.com/api/v4/internal/org_mover/manifest"

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Generated by Claude Opus 5.5 in Claude Code (T3 Code)

Session baef61c8-20b4-40f1-9267-f27e9e05f8ae

Edited by Michael Kozono

Merge request reports

Loading
Loading