<!-- MAINTENANCE RULE — NEVER ADD GITLAB USER HANDLES (`at-username`) HERE. This description is re-synced on every tracking update. Each sync re-notifies every user referenced by handle, creating chronic notification spam. Use roles/teams ("Package team", "Geo reviewer") or link to comment URLs instead. Enforced by the protocells-geo-sync-tracking skill's pre-sync grep. --> **⚠️ AUTO-GENERATED:** Do not edit this description directly. Source of truth: [tracking_issue_description.md](https://gitlab.com/cwoolley-gitlab/ai-workflow-artifacts/-/blob/master/workflow-artifacts/workitem-19604/protocells_geo_support_workflow/tracking_issue_description.md). Any manual edits **will be overwritten** on next sync. # Package Stage: Protocells Support - Tracking ## Work Items and Repos: Tracking, Implementation, and Testing - **Protocells Support Epic:** https://gitlab.com/groups/gitlab-org/-/work_items/19604+s - **Infrastructure Team Protocells Dependency Reporting Epic:** https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1721+s - **AI-driven plan:** https://gitlab.com/cwoolley-gitlab/ai-workflow-artifacts/-/blob/master/workflow-artifacts/workitem-19604/a-plan/v41/plan-v41.md - **AI-driven implementation automation skills:** https://gitlab.com/cwoolley-gitlab/ai-workflow-artifacts/-/tree/master/workflow-artifacts/workitem-19604/protocells_geo_support_workflow/claude-code-skills - **AI-driven verification and testing environment:** https://gitlab.com/issue-reproduce/pkg-geo-get — see [docs/verification-process.md](https://gitlab.com/issue-reproduce/pkg-geo-get/-/blob/main/docs/verification-process.md?ref_type=heads) - **AI-driven source of truth tracking issue (this issue):** https://gitlab.com/gitlab-org/gitlab/-/work_items/593813+s ## Current Status ### In Progress - GroupComponentFile — Task 1 (DB schema) MR [!235129](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/235129+s) and Task 2 (implementation) MR [!235130](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/235130+s) created; both pipelines went green on 2026-05-16 then accepted a milestone bump to 19.1 on 2026-05-19 (pipelines re-running). Task 2 has `discussions_not_resolved` (review iteration in progress). On 2026-05-20, both MRs received additional ProjectComponentFile-style review fixes: `:with_persisted_chain` factory trait (resolves the NotNullViolation that surfaced once `copy_group_id` was switched to a direct-column read — `build(:debian_group_component_file).save!` cascade-builds the upstream chain unsaved, so the FOSS `set_sharding_key_from_distribution` callback can't populate the file's own `group_id` in memory and the after_save state-row create trips the NOT NULL constraint), removal of redundant `after_save :save_verification_details`, FF-disabled destroy-cascade spec, and `table_size: small` in state db dictionary. Generator updated to emit the same shape going forward (plan v41). GET verification, formal review request, and merge still pending. Task 3 work item 599539 exists; MR not yet created. - GroupDistribution and ProjectDistribution — D-split spike resolved via [issue 592643](https://gitlab.com/gitlab-org/gitlab/-/issues/592643+s) (immutable distribution file tables, create-new-on-change). Foundation MRs: [!226701](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/226701+s) (DB schema, `592643-mr01-db-schema-new-file-tables`, **still open** — `database::review pending`, `workflow::planning`, no human reviewers assigned, `missed:18.10`/`missed:18.11`/`missed:19.0` labels) and [!226788](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/226788+s) (FOSS models + service + API, `592643-mr02-foss-models-service-api`, still open). MR !226788 validated on GET 2026-05-18: pipeline [2529098789](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2529098789) (Ubuntu-24.04-branch passed on attempt 4/5 after three Gitaly rate-limit failures), build `gitlab-ee_18.11.3+rfbranch.2529110606.d70f85a9-0` deployed to both Geo nodes. Automated Geo verification PASS for pre-existing replicators (PackageFile synced 0→1 in 50s; NuGet Symbol synced 0→1 in 40s); Debian distribution / group-distribution / project-component-file replicators correctly SKIP because they're not in this MR's scope (the Geo replicators for the new distribution file tables come in follow-up MRs). **Migration rollback + rollforward PASS** for all three migrations this MR introduces (`20260510120000_add_distribution_file_pair_consistency_check`, `20260510120100_queue_backfill_debian_group_distribution_file_records`, `20260510120101_queue_backfill_debian_project_distribution_file_records`) — `up`/`down` symmetric across `schema_migrations`, `pg_constraint`, and `batched_background_migrations`. Details in `results/226788/2026-05-18-082436-migration-rollback-rollforward.md` of the pkg-geo-get repo. Manual replication walkthrough skipped because the documented flow exercises pre-existing Geo paths that the automated Phase 5 already covered, and the new distribution file API surface this MR adds will be exercised by its callers in the future GroupDistribution / ProjectDistribution Geo replicator MRs. ### In Review - ProjectComponentFile — Task 1 MR !228787 **merged 2026-05-13**. Review iteration in progress on Task 2 MR !228959 (reviewers assigned, `analytics instrumentation::review pending`, `database::review pending`, `workflow::ready for review`, `discussions_not_resolved`). Re-verification PASS on GET 2026-04-23 against MR !228959 @ `3b7d2ac` omnibus (migration + loose-FK cleanup both verified end-to-end). Subsequent pushes on !228959 addressing reviewer feedback: FF-disabled spec, direct-column flip in `copy_project_id`, milestone bump to 19.1. Conflicts introduced by the merge of [!235038](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/235038+s) ("Geo: Replicate Vulnerabilities::Export uploads", merged 2026-05-15, touches overlapping cross-cutting Geo wiring) — **rebase complete** (new tip `17073447`, `has_conflicts: false`, `merge_status: can_be_merged`), but **all prior approvals were reset by the rebase** (0/18 approvals). Now in another re-test + re-approval cycle. ### On Hold - RPM: Package team confirmed RPM registry is an abandoned MVC. Geo replication on hold per the Package team ([comment](https://gitlab.com/gitlab-org/gitlab/-/issues/379055#note_3220415029)). Tasks and MRs closed. ### Infrastructure Blocker — gitlab.com gitaly rate-limit on omnibus builds (2026-04-23, updated 2026-05-19) - **Impact on this workflow:** On 2026-04-23, multiple omnibus-gitlab-mirror build jobs on MR !228959 failed during `git clone gitlab-org/gitlab.git` with `fatal: remote error: GitLab is currently unable to handle this request due to load`. Multiple retries also failed. This is the same class of failure tracked in https://gitlab.com/gitlab-org/omnibus-gitlab/-/work_items/9776. Each failed omnibus cycle adds ~35-45 minutes per retry, significantly slowing GET verification cadence (often adding hours of wait time before an omnibus build is usable). **Still affecting builds as recently as 2026-05-18** — MR !226788's omnibus required 4 attempts on [pipeline 2529098789](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2529098789) before passing on the Ubuntu-24.04 branch. - **Upstream fix in flight:** Capability MR https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/9365+ ("Always auth for gitlab.com clone/fetch operations") **merged 2026-05-11**. Follow-up MR https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/9411+ ("Enable `FF_USE_GIT_PROACTIVE_AUTH`") **still open**. Until the feature flag is flipped on in the `gitlab-org/build/omnibus-gitlab-mirror` pipeline path, every Protocells omnibus build remains exposed to rate-limit flakes. omnibus-gitlab#9776 still open. - **Local mitigation:** Developers can set `proactiveAuth = basic` in their own `~/.gitconfig` for HTTPS calls to `https://gitlab.com` — see https://gitlab.com/gitlab-org/gitlab/-/work_items/591939#note_3143019683. Workaround captured as snippet for re-use across dev machines. - **Workflow note:** Future Protocells model MRs that require omnibus builds should expect this flake until !9411 merges and the `FF_USE_GIT_PROACTIVE_AUTH` feature flag is enabled for the mirror pipeline. ## Process per Add Geo Support Issue **Entry point:** Run `/protocells-geo-next` to see what's actionable and pick the next step. For each issue, the cycle is ([Per-Model Execution Checklist](https://gitlab.com/cwoolley-gitlab/ai-workflow-artifacts/-/blob/master/workflow-artifacts/workitem-19604/a-plan/v41/plan-v41.md#per-model-execution-checklist)): 1. [Validate generator](https://gitlab.com/cwoolley-gitlab/ai-workflow-artifacts/-/blob/master/workflow-artifacts/workitem-19604/a-plan/v41/plan-v41.md#c1-pre-flight-validate-generator) against Nuget::Symbol reference 2. [Generate Task 1](https://gitlab.com/cwoolley-gitlab/ai-workflow-artifacts/-/blob/master/workflow-artifacts/workitem-19604/a-plan/v41/plan-v41.md#c2-task-1-modify-db-schema-for-geo-support--mr-1) (DB schema) → create MR 3. [Generate Task 2](https://gitlab.com/cwoolley-gitlab/ai-workflow-artifacts/-/blob/master/workflow-artifacts/workitem-19604/a-plan/v41/plan-v41.md#c3-task-2-implement-geo-replication-behind-feature-flag--mr-2) (implementation) → create stacked MR 4. [Deploy, verify, and iterate](https://gitlab.com/issue-reproduce/pkg-geo-get/-/blob/main/docs/verification-process.md) — omnibus build → deploy to GET → automated verification → manual verification. Iterate as needed: fix issues → re-generate → rebuild → re-verify. See also [review iteration protocol](https://gitlab.com/cwoolley-gitlab/ai-workflow-artifacts/-/blob/master/workflow-artifacts/workitem-19604/a-plan/v41/plan-v41.md#c8-review-iteration-protocol). 5. When both MRs are verified and review-ready, request review 6. After Task 1 and Task 2 merge: [Task 3](https://gitlab.com/cwoolley-gitlab/ai-workflow-artifacts/-/blob/master/workflow-artifacts/workitem-19604/a-plan/v41/plan-v41.md#c4-task-3-release-geo-support--mr-3) (release) → [feature flag rollout](https://gitlab.com/cwoolley-gitlab/ai-workflow-artifacts/-/blob/master/workflow-artifacts/workitem-19604/a-plan/v41/plan-v41.md#c7-feature-flag-rollout-process-per-issue-after-task-3) Task 1 is never merged independently — both Task 1 and Task 2 must be fully verified before requesting review on either. ## Prerequisites - [x] A1: Dual-uploader question resolved → implementation in https://gitlab.com/gitlab-org/gitlab/-/issues/592643+s - [x] A2: Organization-based selective sync validated - [x] A3: Object storage handling decided for package models ## Workflow Tooling - [x] B2: Generator script created and validated - [x] B3: Claude Code skills created and symlinked - [x] B4: This tracking issue created - [x] B5: Geo tracking database configured for development (via `geo.experimental.allow_secondary_tests_in_primary`) ## Add Geo Support Issues ### 1. Packages::Debian::ProjectComponentFile (https://gitlab.com/gitlab-org/gitlab/-/issues/333611+s) - [x] Generate Task 1 (DB schema) → https://gitlab.com/gitlab-org/gitlab/-/work_items/594713+s - [x] MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/228787+s - [x] Generate Task 2 (implementation) → https://gitlab.com/gitlab-org/gitlab/-/work_items/594714+s - [x] MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/228959+s - [x] Verification passed - 2026-03-30: Discovered llhttp-ffi bug blocking ALL Geo blob sync (#595139+s) — runtime workaround (gem upgrade + monkey-patch) required on both nodes until MR !229530+s merges - 2026-03-30: Debian ProjectComponentFile replication verified via S3 bucket content match — PASS, but REST API counters absent (replicator not registered in REPLICATOR_CLASSES) - 2026-03-31: Omnibus build green, deployed to GET, 0 Geo-related CI failures - 2026-04-07: **Automated verification PASS.** With llhttp workaround + `geo_packages_debian_project_component_file_replication` feature flag enabled, REST counter incremented 4→5 in ~50s. Root cause of llhttp issue: `rugged` 1.9.0 corrupts Omnibus libffi 3.2.1 FFI callbacks on kernel >= 6.8 (#595139+s). Upstream fix: MR !230361+s. - 2026-04-13: **Manual verification PASS.** GUI inspection confirmed replication working correctly. - 2026-04-22: **Task 1 + Task 2 MRs backfilled** with loose-FK support after informal pre-review ([comment](https://gitlab.com/gitlab-org/gitlab/-/work_items/333611#note_3273035894)). Generator updated to emit `config/gitlab_loose_foreign_keys.yml` entry + `track_record_deletions` post-migration + removal of hard FK from state→parent. Prior verification is now stale; need fresh omnibus build → GET deploy → automated + manual verification cycle before requesting review. - 2026-04-23: **Re-verification PASS on MR !228959 @ `3b7d2ac`** (omnibus [job 14047100437](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/jobs/14047100437) from [pipeline 2472582935](https://gitlab.com/gitlab-org/gitlab/-/pipelines/2472582935), deployed to GET after `terraform taint` + `apply` on both rails nodes): automated (`verify-metadata-replication` debian-project-component-file PASS synced 0→1 in 30s) + manual (secondary S3 bucket `pkg-geo-secondary-packages` confirmed, REST counters count=synced=verified=1, failed=0) + **loose-FK cleanup PASS** (destroying parent on primary → `LooseForeignKeys::DeletedRecord` created → `CleanupWorker` → state row gone on both sites, registry row cleaned from secondary tracking DB). Details in `results/228959/2026-04-23-035104-manual-verification-and-loose-fk.md` of the pkg-geo-get repo. - 2026-05-20: **Factory chain fix on MR !228959 (commit `d98917ec`).** State model `copy_project_id` now reads the file's own column directly (was a 2-hop traversal). To make this work in tests, added a `:with_persisted_chain` FOSS factory trait that forces the upstream chain to be created (cascade-build leaves `distribution.project_id` nil at the file's before_validation, so the FOSS `set_sharding_key_from_distribution` callback can't populate the file's own column in memory, and the after_save callback creating the state row trips the NOT NULL constraint). Verifiable-model shared example opts into the trait. Generator updated to mirror this (plan v41). Pre-merge re-verification not re-run — change is test-side + a code-side equivalence (direct column = trigger-populated column the 2-hop walk used to read). - [x] Request review — **Status: !228787 merged 2026-05-13. Review iteration in progress on !228959 (reviewers assigned, `database::review pending`, `analytics instrumentation::review pending`, `workflow::ready for review`, `discussions_not_resolved`). Recent pushes addressing reviewer feedback on !228959: added FF-disabled spec (`35d90fa1e520`, addresses [note 3347677506](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/228959#note_3347677506)), flipped `copy_project_id` to direct column (`103076002b9d`, addresses [note 3358355513](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/228959#note_3358355513)), bumped milestone to 19.1 (`6317d9a6e06a`), then on 2026-05-19/20 reverted-and-reapplied `copy_project_id` with a paired `:with_persisted_chain` factory trait fix (`d98917ec`, addresses [note 3363275181](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/228959#note_3363275181)). Conflicts introduced by the merge of [!235038](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/235038+s) ("Geo: Replicate Vulnerabilities::Export uploads", merged 2026-05-15) — **rebase complete** (new tip `17073447`, `merge_status: can_be_merged`), but **all prior approvals were reset by the rebase** (0/18). Now in another re-test + re-approval cycle.** - [ ] Task 1 and Task 2 merged - [x] Generate Task 3 (release) → https://gitlab.com/gitlab-org/gitlab/-/work_items/594715+s - [ ] MR: _not yet created_ - [ ] Feature flag rollout ### 2. Packages::Debian::GroupComponentFile (https://gitlab.com/gitlab-org/gitlab/-/issues/556945+s) - [x] Generate Task 1 (DB schema) → https://gitlab.com/gitlab-org/gitlab/-/work_items/599537+s - [x] MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/235129+s — **Status: prior pipeline green; milestone bumped to 19.1 on 2026-05-19, new pipeline running; on 2026-05-20 also rebased to pick up `table_size: small` in state db dict; no conflicts; awaiting review** - [x] Generate Task 2 (implementation) → https://gitlab.com/gitlab-org/gitlab/-/work_items/599538+s - [x] MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/235130+s — **Status: prior pipeline green 2026-05-16; milestone bumped to 19.1 on 2026-05-19, new pipeline running; conflicts from [!235038](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/235038+s) merge resolved via rebase; on 2026-05-20 received `:with_persisted_chain` factory trait fix (resolves NotNullViolation surfaced by the verifiable-model shared example), removal of redundant `after_save :save_verification_details`, and FF-disabled destroy-cascade spec (new tip `259895e9`, `has_conflicts: false`); `discussions_not_resolved` (review iteration in progress)** - [ ] Verification passed — **Status: in progress — pipelines re-running after 2026-05-20 fixes. Once green, run omnibus build → deploy to GET → automated + manual verification.** - [ ] Request review - [ ] Task 1 and Task 2 merged - [x] Generate Task 3 (release) → https://gitlab.com/gitlab-org/gitlab/-/work_items/599539+s - [ ] MR: _not yet created_ - [ ] Feature flag rollout ### 3. Packages::Debian::GroupDistribution (https://gitlab.com/gitlab-org/gitlab/-/issues/556947+s) - [ ] Generate Task 1 (DB schema) — **Status: D-split spike resolved via [issue 592643](https://gitlab.com/gitlab-org/gitlab/-/issues/592643+s); waiting on foundation MRs [!226701](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/226701+s) (DB schema) and [!226788](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/226788+s) (FOSS models + service + API) to merge before this can be generated** - [ ] MR: _not yet created_ - [ ] Generate Task 2 (implementation) - [ ] MR: _not yet created_ - [ ] Verification passed - [ ] Request review - [ ] Task 1 and Task 2 merged - [ ] Generate Task 3 (release) - [ ] MR: _not yet created_ - [ ] Feature flag rollout ### 4. Packages::Debian::ProjectDistribution (https://gitlab.com/gitlab-org/gitlab/-/issues/556946+s) - [ ] Generate Task 1 (DB schema) — **Status: D-split spike resolved via [issue 592643](https://gitlab.com/gitlab-org/gitlab/-/issues/592643+s); waiting on foundation MRs [!226701](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/226701+s) (DB schema) and [!226788](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/226788+s) (FOSS models + service + API) to merge before this can be generated** - [ ] MR: _not yet created_ - [ ] Generate Task 2 (implementation) - [ ] MR: _not yet created_ - [ ] Verification passed - [ ] Request review - [ ] Task 1 and Task 2 merged - [ ] Generate Task 3 (release) - [ ] MR: _not yet created_ - [ ] Feature flag rollout ### 5. Packages::Rpm::RepositoryFile (https://gitlab.com/gitlab-org/gitlab/-/issues/379055+s) — **ON HOLD** RPM registry upload was never implemented (abandoned mid-2022 MVC). Package team confirmed Geo replication on hold until the format is revisited. Prod RPM tables are empty (0 rows). Tasks closed, MRs closed. - [x] Generate Task 1 (DB schema) → https://gitlab.com/gitlab-org/gitlab/-/work_items/593815+s (closed) - [x] MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/227615+s (closed) - [x] Generate Task 2 (implementation) → https://gitlab.com/gitlab-org/gitlab/-/work_items/593816+s (closed) - [x] MR: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/227707+s (closed) - [x] Generate Task 3 (release) → https://gitlab.com/gitlab-org/gitlab/-/work_items/593817+s (closed) - [x] Update `every_gitlab_uploader_spec.rb` to mark RPM replication as not needed → https://gitlab.com/gitlab-org/gitlab/-/merge_requests/230471+s (merged 2026-04-09)