## Why The GitHub continuous sync feature exists as a working proof of concept in merge request [!251883](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/251883) (`Draft: POC - Import GitHub repositories with continuous sync`, branch `feature/github-connected-project-poc`). It is about 40,000 lines, was never code-reviewed, and a real bug was already found in it. That MR is closed and was never merged. We are porting its pieces into `master` as small merge requests that a reviewer can actually read, all behind the `github_continuous_import` feature flag (type `wip`, default off). This issue ports the engine that keeps the GitLab copy current after the initial import: it re-fetches branches and tags from GitHub, and re-applies GitHub issues, pull requests, and comments to their GitLab records. LFS objects are not synced here; https://gitlab.com/gitlab-org/gitlab/-/work_items/633491 adds that. Webhooks (https://gitlab.com/gitlab-org/gitlab/-/work_items/628589), the stop/cutover flow (https://gitlab.com/gitlab-org/gitlab/-/work_items/628590), and read-only enforcement all sit on top of it. Drift is tracked with two counters on each `Import::Sync::RepositoryState` row (the PoC calls it `Continuity::RepositorySyncState`), one row per data domain (`inventory`, `source`, `work_items`). `desired_generation` goes up whenever something says "a sync is needed": a webhook, the 30-minute repair cron, a manual "Update now", or the handoff after the initial import. `synced_generation` goes up only after a full pass for that domain succeeds. The invariant is `synced_generation <= desired_generation`, and both counters only move forward, under a lock. The model already validates this (`synced_generation_does_not_exceed_desired_generation`, landed in https://gitlab.com/gitlab-org/gitlab/-/work_items/628581); this issue lands the code that has to respect it. ## What to do Read the files below and port them into a fresh MR against `master`, renaming `Continuity::` to `Import::Sync::` (so `Continuity::Repositories::ReconcileSourceService` becomes `Import::Sync::Repositories::ReconcileSourceService`, under `ee/app/services/import/sync/repositories/` (the lease primitive from https://gitlab.com/gitlab-org/gitlab/-/work_items/628582 is under `app/`); same for workers, under `ee/app/workers/import/sync/repositories/`). Rename Redis lease keys, advisory lock namespaces (`continuity:...`), queue names, and the table names in `defer_on_database_health_signal` to match. **Services.** `ReconcileSourceService` fetches GitHub identity plus branches and tags for one repository under an `ApplicationLease`, a row lock, and a generation check (`verify_application_fence!`), and advances `synced_generation` only when everything succeeded. Its `#fetch_source` writes refs with a bare `Repository#fetch_as_mirror` and, as its own docstring says, "never ... invokes post-receive processing". That means synced branch updates do not trigger CI/CD pipelines today. This is a known, confirmed gap that https://gitlab.com/gitlab-org/gitlab/-/work_items/628597 fixes; port it as-is and do not fix it here. Port one small addition to `#fetch_source` beyond the PoC: re-check `application_generation` immediately before it writes refs. In the PoC, `ApplicationMutationBarrier.with_shared` stayed held by `ReconcileWorker` for the whole reconcile pass, including the ref write, so a stale generation couldn't sneak in between `verify_application_fence!` and `fetch_source`. That barrier isn't ported (see "Fencing" below), so without a fresh check right before the write, a repository that was disconnected or cut over in the gap between the earlier fence check and `apply_provider_identity`/`fetch_source` could still get a ref written to it. Add the re-check immediately before the `fetch_as_mirror` call, not just earlier in the method. https://gitlab.com/gitlab-org/gitlab/-/work_items/628597 keeps this same re-check when it changes how refs are applied through `ff_merge`/`Branches::CreateService`. `ReconcileWorkItemsService` converges issues, pull requests, and comments page by page with a small JSON cursor on `RepositoryState#cursor`, at most `MAX_PROVIDER_PAGES_PER_INVOCATION = 2` pages per run and a 5-minute `SAFETY_OVERLAP` window, clearing the cursor and advancing `synced_generation` only after a whole pass. Settled in the 2026-09-10 scope audit: drop the cursor's compare-and-swap (`persist_cursor!`'s `expected_cursor:` parameter and `same_cursor_pass?` check) and its `CURSOR_VERSION` field; the lease `verify!`, the repository row lock, and the `application_generation` check that already guard every cursor write already exclude a concurrent writer, so the CAS is redundant. Keep the resumable cursor itself, `MAX_PROVIDER_PAGES_PER_INVOCATION`, and `SAFETY_OVERLAP`. `ScheduleReconcileService` is the shared "mark these domains as needing sync" transaction: it locks the rows and bumps `desired_generation`; it never talks to GitHub or enqueues jobs. `StartCatchUpService` is the one-time, idempotent handoff from initial import to sync: under the lease and a row lock, it flips `authority_state` to `provider_canonical` when still `provisioning?` and enqueues `ReconcileWorker`. It no longer replays buffered webhooks (https://gitlab.com/gitlab-org/gitlab/-/work_items/628589 drops the webhook delivery table that replay read from, see "Forward references and the flag" below), so it becomes DB-only plus the one enqueue. `RequestUpdateService` is the "Update now" entry point. **One adaptation.** `WorkItems::ReconcileObjectService` is the only production file that reads the old polymorphic mapping. PoC lines 125-131: ```ruby if mapping unless mapping.local_record_type == expected_type raise MappingConflictError, 'GitHub object mapping has the wrong GitLab record type' end return mapping.local_record end ``` https://gitlab.com/gitlab-org/gitlab/-/work_items/628581 replaced `local_record_type`/`local_record_id` on `Import::Sync::ExternalObject` with nullable `issue_id`, `merge_request_id`, `note_id` columns; `local_record` returns whichever is set and `record!`/`for_local_record` kept their signatures. Rewrite the type check against the new columns (or the class of `local_record`) and leave the rest alone. Also settled in the 2026-09-10 scope audit: drop `ReconcileObjectService#fingerprint` and the `fingerprint:` keyword it passes to `ExternalObject.record!`. It's write-only in the PoC; drift detection already compares live attributes (`local_attributes_match?`), so `external_objects.fingerprint` (not ported by https://gitlab.com/gitlab-org/gitlab/-/work_items/628581) has no reader. **Errors and logging.** Settled in the same audit: collapse the roughly 45 per-service `Class.new(StandardError)` declarations across this layer into one `Import::Sync::Errors` module, and add `Gitlab::Import::Logger` calls at suspend, restore, disconnect, and stop transitions; the PoC logs nothing in `app/services/continuity`. For this issue that means the transitions inside `StartCatchUpService` (provisioning to provider_canonical) and the failure paths in `ReconcileSourceService`/`ReconcileWorkItemsService`. **Fencing.** `ApplicationMutationBarrier` and `BootstrapMutationBarrier` are not ported; neither is landed by https://gitlab.com/gitlab-org/gitlab/-/work_items/628582 or anywhere else. Both are PostgreSQL advisory locks held across a transaction and network calls to GitHub, and GitLab requires PgBouncer in transaction pooling mode (`doc/administration/postgresql/pgbouncer.md`, line 247), under which a session lock taken on one backend connection can't reliably be released on another and can leak. `ReconcileWorker` takes `ApplicationMutationBarrier.new(repository.id).with_shared` around its whole reconcile pass in the PoC; drop that call and keep only the `ApplicationLease` (landed in https://gitlab.com/gitlab-org/gitlab/-/work_items/628582, a dependency of this issue) around the same pass. The barrier's job, keeping a stop/disconnect from proceeding while a shared holder is mid-pass, is replaced by each service's own generation check, plus the new pre-write check on `ReconcileSourceService#fetch_source` described above. **Workers.** `ReconcileWorker` (`data_consistency :sticky`, `idempotent!`, `deduplicate :until_executing, if_deduplicated: :reschedule_once, including_scheduled: true`, `retry: 5, dead: false`) runs both reconcilers under one lease (not a shared barrier, see "Fencing" above) and reschedules itself on follow-up. Settled in the 2026-09-10 scope audit: drop the manual `rescue Gitlab::ExclusiveLeaseHelpers::FailedToObtainLockError; self.class.perform_in(10.seconds, ...)` re-enqueue on lease contention; let Sidekiq's own retry (or the lease's own retries) handle it instead. Note that `ReconcileWorker`'s `deduplicate :until_executing, including_scheduled: true` could swallow a follow-up enqueue that arrives while a scheduled retry is already queued; the engineer landing this should pick dedup settings that don't drop that follow-up. `ScheduleRepairWorker` (`urgency :low`) calls `RequestUpdateService`. Settled in the same audit: it no longer redrives buffered webhook deliveries (drop `redrive_webhook_deliveries` and `MAX_WEBHOOK_DELIVERY_REDRIVES`); duplicates are harmless under the no-persistence webhook design https://gitlab.com/gitlab-org/gitlab/-/work_items/628589 lands, so there is nothing left to redrive. Add the `all_queues.yml` and `sidekiq_queues.yml` entries for these two workers (PoC: urgency low, weight 1, idempotent; `has_external_dependencies: true` for `ReconcileWorker`, false for `ScheduleRepairWorker`) and confirm them against `doc/development/sidekiq/` rather than copying, since `ReconcileWorker` calls GitHub and Gitaly. `ScheduleRepairsCronWorker`, the cron that fans out to both of these workers plus the two https://gitlab.com/gitlab-org/gitlab/-/work_items/628589/https://gitlab.com/gitlab-org/gitlab/-/work_items/628590 workers, lands in https://gitlab.com/gitlab-org/gitlab/-/work_items/628590 instead, once every fan-out target it calls actually exists. **Forward references and the flag.** This layer lands before https://gitlab.com/gitlab-org/gitlab/-/work_items/628589. With the webhook-delivery redrive dropped from both `StartCatchUpService` and `ScheduleRepairWorker` (see above), those two no longer reference anything from https://gitlab.com/gitlab-org/gitlab/-/work_items/628589. One forward reference remains: `reconcile_worker.rb:33` still rescues `Gitlab::GithubImport::RepositoryNotSelectedError` by calling `DisconnectForProviderAccessLossService` (https://gitlab.com/gitlab-org/gitlab/-/work_items/628589). Ruby resolves constants lazily, so the file loads, but reaching that line raises `NameError`. The worker doesn't check the feature flag today. Make that path unreachable while the flag is off (the cheapest option is a `Feature.enabled?(:github_continuous_import)` guard at the top of `ReconcileWorker#perform`), and say in the MR description what you chose. **Simulator.** None of these production files has a `test_mode?` branch and none of the specs below reference the offline GitHub simulator; confirm that while porting. If any do, use stubbed GitHub responses and local git fixtures instead, as the existing importer specs do. **Lease renewal points.** https://gitlab.com/gitlab-org/gitlab/-/work_items/628582 lands the lease with a `Context#renew!` method and no heartbeat thread, so each service here must renew after every round trip to GitHub or Gitaly: `ReconcileSourceService` right after the git fetch returns and before it applies or verifies refs; `ReconcileWorkItemsService` after each page of API results. Call `verify!` (or `verify_repository!`) immediately before every write, as the PoC already does. **User mapping must keep working during sync.** Improved user contribution mapping is a requirement for synced projects, not an option. The PoC handles it and the port must preserve it: `WorkItems::ReconcileObjectService` calls `importer.push_placeholder_references(record)` both when it creates a record and when it updates one (and re-pushes for an already-current record on a page retry, so Redis entries cannot expire during a long retry); `ReconcileWorkItemsService#load_placeholder_references!` then drains the project's placeholder store through `Import::PlaceholderReferences::LoadService` once per page, bounded by `MAX_PLACEHOLDER_BATCHES_PER_PAGE = 1`, and fails the page without advancing the cursor if loading fails. The PoC specs for this are `reconcile_object_service_spec.rb` ("re-pushes placeholder references for an already-current provider version") and `reconcile_work_items_service_spec.rb` (the "when reconciliation creates placeholder-user references" context, "continues after a bounded placeholder-reference batch", "creates the issue and durably loads its placeholder reference"). Port them unchanged in behavior. ## PoC files to read Production: `app/services/continuity/repositories/{reconcile_source_service,reconcile_work_items_service,schedule_reconcile_service,start_catch_up_service,request_update_service}.rb`, `app/services/continuity/repositories/work_items/reconcile_object_service.rb`, `app/workers/continuity/repositories/{reconcile_worker,schedule_repair_worker}.rb`, and the `ReconcileWorker`/`ScheduleRepairWorker` entries in `config/sidekiq_queues.yml`, `app/workers/all_queues.yml`. Specs: `spec/services/continuity/repositories/{reconcile_source_service,reconcile_work_items_service,schedule_reconcile_service,start_catch_up_service,request_update_service}_spec.rb`, `spec/services/continuity/repositories/work_items/reconcile_object_service_spec.rb`, `spec/workers/continuity/repositories/{reconcile_worker,schedule_repair_worker}_spec.rb`. The other specs in those directories, and the fencing-primitive specs, belong to https://gitlab.com/gitlab-org/gitlab/-/work_items/628582, https://gitlab.com/gitlab-org/gitlab/-/work_items/628589, and https://gitlab.com/gitlab-org/gitlab/-/work_items/628590. Drop the CAS-specific example in `reconcile_work_items_service_spec.rb`, `'does not overwrite a newer page from another same-pass worker'`, which asserts on `expected_cursor:`/`same_cursor_pass?` behavior that no longer exists; replace it, if needed, with a spec that a stale `application_generation` (not a stale cursor read) is what stops a superseded writer from checkpointing. ## Acceptance criteria - Everything above exists under `Import::Sync::Repositories::` with passing specs. - `WorkItems::ReconcileObjectService` has no reference to `local_record_type`, `local_record_id`, or `fingerprint`. - No `test_mode?` branch in ported code; no spec depends on the simulator. - Queue, urgency, data consistency, and external-dependency declarations are confirmed against `doc/development/sidekiq/`. - The forward reference to `DisconnectForProviderAccessLossService` (https://gitlab.com/gitlab-org/gitlab/-/work_items/628589) in `ReconcileWorker` cannot be reached while the flag is off, and the MR description says how. `StartCatchUpService` and `ScheduleRepairWorker` have no forward reference to https://gitlab.com/gitlab-org/gitlab/-/work_items/628589 left, since neither redrives webhook deliveries anymore. - `ReconcileWorkItemsService`'s cursor has no `expected_cursor:`/`same_cursor_pass?` compare-and-swap and no `CURSOR_VERSION` field; the reviewer has confirmed the lease, row lock, and generation check are sufficient without it. - The reviewer has traced, in writing on the MR, why `synced_generation` cannot pass `desired_generation` or move backward when a second job bumps `desired_generation` during a pass, and why the pre-write generation check immediately before `fetch_source`'s `fetch_as_mirror` call closes the gap left by not holding `ApplicationMutationBarrier` for the whole pass. - `Import::Sync::Errors` replaces the per-service `Class.new(StandardError)` declarations in this layer, and `Gitlab::Import::Logger` calls exist at the suspend/restore/disconnect/stop transitions this issue owns. - Placeholder references are pushed on every sync create and update and loaded once per page; the PoC specs named above pass. - A spec exists for a source user reassigned to a real user between two sync passes: the next update attributes the change to the real user and pushes no new placeholder reference (`Import::PlaceholderReferences::Pusher#push_reference` already skips records mapped to a human; the spec confirms the sync path goes through it). Add this spec if the PoC has none. ## Out of scope - The `ApplicationLease` primitive itself (`ApplicationLease`, the `sleeping_lock.rb` change); https://gitlab.com/gitlab-org/gitlab/-/work_items/628582 lands it, this issue only calls it. - `ApplicationMutationBarrier` and `BootstrapMutationBarrier`. Neither is ported by https://gitlab.com/gitlab-org/gitlab/-/work_items/628582 or any other issue; see "Fencing" above for what replaces them here. - Webhooks, `DisconnectForProviderAccessLossService`, `LifecycleRepairWorker` (https://gitlab.com/gitlab-org/gitlab/-/work_items/628589). `Continuity::WebhookDelivery`, `ProcessWebhookDeliveryService`, and `WebhookDeliveryWorker` are dropped from the port entirely (https://gitlab.com/gitlab-org/gitlab/-/work_items/628589); this issue never references them. - Stop/cutover services, `FinishStopWorker`, and `ScheduleRepairsCronWorker` (https://gitlab.com/gitlab-org/gitlab/-/work_items/628590). - Fixing the CI/CD trigger gap (https://gitlab.com/gitlab-org/gitlab/-/work_items/628597). - LFS objects for synced refs: `ReconcileLfsObjectsService` and the importer and download-service changes it needs are tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/633491. `ReconcileSourceService` is already on `master` without an LFS call (from !259782). The PoC's `reconcile_source_service.rb` and its spec, named in the file lists above, carry the LFS call, the early return for an incomplete LFS page, two LFS examples, and LFS wiring in the spec's shared setup (`let(:lfs_reconcile_service)`, the `ReconcileLfsObjectsService.new` stub, one assertion); leave those out, that issue adds them. Until it lands, `ReconcileSourceService` fetches refs but not the LFS objects they point to. - Redesigning the lease/generation scheme, or the offline simulator. - The accepted window this creates: a worker paused past its lease that wakes up after a cutover or disconnect could still write a ref in the brief gap between its pre-write check and the actual `fetch_as_mirror` call. This is a known, accepted Experiment limitation, tracked alongside the "no atomic snapshot" caveat in https://gitlab.com/gitlab-org/gitlab/-/work_items/628600; it is not fixed here. ## Notes for the reviewer - Review lens: the generation invariant under lease and row-lock ordering, against a concurrent-job scenario, including the new pre-write check in `#fetch_source`. Generation assertions exist in `reconcile_source_service_spec.rb`, `reconcile_work_items_service_spec.rb`, `schedule_reconcile_service_spec.rb`, and `request_update_service_spec.rb` (they bump `desired_generation` mid-run), but no spec runs two jobs at once, and the worker specs do not touch generations at all. If reasoning through the code is not enough, ask for a two-`ReconcileWorker` spec in this MR. - Read `#fetch_source` knowing the pipeline gap is real and deliberately left for https://gitlab.com/gitlab-org/gitlab/-/work_items/628597. LFS objects are not downloaded in this pass either; see "Out of scope". --- **OAuth-path note:** The reconciliation services here operate on `RepositoryState`'s generation counters, not on any provider-specific field. No change needed for OAuth support (see epic 23553, now pursuing GitHub OAuth as the active connection path).