### Summary Two bugs, both triggered when something moves within the same top-level group (the Zoekt index does not change): 1. **Subgroup move**: `EE::Groups::TransferService` queues no Zoekt work when the top-level group is unchanged. Every project under the moved subgroup keeps its old `traversal_ids` in the Zoekt shard. The Zoekt repository stays `ready`. 2. **Project move**: `Search::Zoekt::ProjectTransferWorker` always queues a delete for the old namespace and an index for the new one. Within one top-level group both target the same Zoekt repository. The delete runs, its callback destroys the repository row, and the index task is either `skipped` (the repository is no longer indexable) or removed along with the row. The old shard stays on disk until the indexer's deferred delete removes it. In both cases, group-level search at the new location (and any new ancestor) misses the project, because group search filters shards by `traversal_ids`. Differences between the two cases: - Subgroup move: project-level search keeps working (it filters by repository id), and search at the top-level group still finds the project, since the old `traversal_ids` still start with the root id. The stale state persists until something else reindexes the project, for example a push. - Project move: the Zoekt repository row is gone, and once the indexer's deferred delete removes the old shard (within 7 minutes in GDK), the project is missing from all code search: the new group, the old group, the top-level group, and the project itself. It stays that way until a push, or until the first project-level search queues `index_async` and repairs that project. Also, in every move that deletes (same-root project move, and moves to another or a non-enrolled top-level group), the old group keeps returning the moved project's code until the deferred delete runs. It is not yet checked whether Rails filters those results before displaying them. Moves across enrolled top-level groups, and moves into an enrolled group from a non-enrolled one, reindex correctly. This is not specific to imports. It was found while investigating a customer who migrated to GitLab.com and reported that group-level search only found repositories committed to since the migration, while project-level search worked. That matches the subgroup-move case; whether they reorganized after migrating is unconfirmed. ### Steps to reproduce Prerequisites: - GDK with Zoekt running. - Exact code search indexing and search enabled. - A top-level group enrolled for exact code search (example: `toolbox`). - Signed in as admin/owner. **A. Subgroup move** 1. In the enrolled top-level group, create subgroups `zoekt-a` and `zoekt-b`. 2. In `zoekt-a`, create project `move-check` with "Initialize repository with a README". 3. Edit the README, add a unique word such as `zoektmovecheck`, commit. 4. Wait about a minute. On the Code tab, search `zoektmovecheck` in the project and in `zoekt-a`. Both find the README. 5. `zoekt-a` > Settings > General > Advanced > Transfer group: choose `zoekt-b`, confirm. 6. Wait a few minutes, search again (Code tab) in the project, `zoekt-b/zoekt-a`, `zoekt-b`, and the top-level group. Result: the project and the top-level group find it; `zoekt-b/zoekt-a` and `zoekt-b` return nothing. **B. Project move** 1. In the enrolled top-level group, create subgroups `zoekt-c` and `zoekt-d`. 2. In `zoekt-c`, create project `move-check-2` with a README, commit a unique word such as `zoektprojectmove`, wait a minute, confirm Code search in `zoekt-c` finds it. 3. Project > Settings > General > Advanced > Transfer project: choose `zoekt-d`, confirm. 4. Wait about a minute. Search (Code tab) in `zoekt-d` first, then `zoekt-c`, then the top-level group. Do not search in the project yet: a project-level search re-queues indexing and hides the bug. Result: `zoekt-d` returns nothing; `zoekt-c` (the old group) and the top-level group still return the file. 5. Wait about 7 minutes and repeat the three group searches. All three return nothing: the deferred delete has removed the old shard and nothing re-created it. 6. Optional: search in the project. It returns nothing the first time, then repairs itself. ### What is the current *bug* behavior? - Subgroup move within the same top-level group: no Zoekt task is created; nothing picks it up later. The moved project's shard keeps its old `traversal_ids`, so it disappears from search at the new subgroup and new parent group, while still appearing at the top-level group and at the project level. - Project move within the same top-level group: the delete task destroys the Zoekt repository row before the index task can use it, so the index task is skipped or dropped. The project disappears from search at the new location straight away. The old group and the top-level group keep returning it until the indexer's deferred delete runs; after that it is missing from all code search until a push or a project-level search re-queues indexing. ### What is the expected *correct* behavior? After any transfer, the affected projects are indexed with their new `traversal_ids`, appear in group search at the new location, and disappear from the old group. ### Relevant logs and/or screenshots <details> <summary>GDK reproduction (master at 75dfa40c9ff9)</summary> All moves were run together by a Rails script, each project with its own unique token. State was checked 90 seconds and 7 minutes after the moves. Project-level search was only run at the end, because it re-queues indexing. **At 90 seconds:** | Case | Move | Zoekt tasks after the move | Zoekt repository | Shard `traversal_ids` | Search at new location | Search at old group | |---|---|---|---|---|---|---| | p1a | project, subgroup -> subgroup, same root | none yet (`ProjectTransferWorker` is throttled) | `ready`, not updated | old | 0 | 1 | | p1b | same | delete `done`, index `skipped` | deleted | old | 0 | 1 | | p1c | same | delete `done`, no index task | deleted | old | 0 | 1 | | p2 | project, subgroup -> top-level group, same root | delete `done`, index `skipped` | deleted | old | found at top level only | 1 | | g1 | subgroup -> other subgroup, same root | none | `ready`, not updated | old | 0 (subgroup and new parent) | n/a | | p3 | project -> another enrolled top-level group | delete + index `done` | `ready` in new index | new | 1 | 1 (old shard until deferred delete) | | g2 | subgroup -> another enrolled top-level group | delete + index `done` | `ready` in new index | new | 1 | n/a | | p4 | project -> non-enrolled top-level group | delete `done` | deleted (expected) | old, pending delete | 0 (expected) | 1 | | p5 | project from non-enrolled group -> enrolled group | index `done` | `ready` | new | 1 | n/a | **At 7 minutes (end state):** | Case | Zoekt tasks after the move | Zoekt repository | Shard on disk | Group search (new / old / top level) | Project search | |---|---|---|---|---|---| | p1a, p1b, p1c | delete `done`; index `skipped` (p1a, p1b) or no index task (p1c) | deleted | none | 0 / 0 / 0 | 0 | | p2 | delete `done`, index `skipped` | deleted | none | top level 0 / old 0 | 0 | | g1 | none | `ready`, not updated | old `traversal_ids` | 0 / n/a / found | found | | p3, g2 | delete + index `done` | `ready` in new index | new only | found / 0 / n/a | found | | p4 | delete `done` | deleted (expected) | none | 0 (expected) | 0 (expected) | | p5 | index `done` | `ready` | new | found | found | So every same-root project move (4 of 4) ended with the project fully missing from Zoekt, and the same-root subgroup move stayed stale. An earlier single run of the subgroup case also stayed stale for the full 3 minutes observed, with no new `zoekt_tasks` rows and no Zoekt jobs queued in Sidekiq. </details> ### Possible fixes Both paths should treat a move within the same top-level group as a metadata change on the same index. 1. Group transfer: queue reindexing for the moved group's projects when the root is unchanged (for example, `force_index_repo` tasks, as `Search::Zoekt::GroupPathChangedEventWorker` creates for path renames). Keep delete-then-index only when the root changes. 2. Project transfer: in `ProjectTransferWorker`, only call `delete_async` when the top-level group changed; otherwise only call `index_async`. A plain `index_repo` is enough, since the indexer forces a rebuild when only metadata changed. Workaround: reindex the affected projects (`force_index_repo`) or push a commit. <details> <summary>Code references</summary> - https://gitlab.com/gitlab-org/gitlab/-/blob/75dfa40c9ff9e0aa37db5c1754c816946f9bc81d/ee/app/services/ee/groups/transfer_service.rb#L136 — Zoekt handling only when `old_root_ancestor_id != project.root_namespace.id`: ```ruby if zoekt_enabled && old_root_ancestor_id != project.root_namespace.id process_zoekt_project(old_root_ancestor_id, old_namespace_had_zoekt_enabled, project) ``` - https://gitlab.com/gitlab-org/gitlab/-/blob/75dfa40c9ff9e0aa37db5c1754c816946f9bc81d/app/services/groups/update_service.rb#L188 — `GroupPathChangedEvent` is published only on path rename, not on transfer. - https://gitlab.com/gitlab-org/gitlab/-/blob/75dfa40c9ff9e0aa37db5c1754c816946f9bc81d/ee/app/workers/search/zoekt/project_transfer_worker.rb#L24-30 — delete for the old namespace, then index, regardless of whether the root changed. - https://gitlab.com/gitlab-org/gitlab/-/blob/75dfa40c9ff9e0aa37db5c1754c816946f9bc81d/ee/app/services/search/zoekt/callback_service.rb#L46-47 — delete callback destroys the repository row. - https://gitlab.com/gitlab-org/gitlab/-/blob/75dfa40c9ff9e0aa37db5c1754c816946f9bc81d/ee/app/models/search/zoekt/task.rb#L52 — index task is `skipped` when the repository is not in `pending`, `initializing`, or `ready`. - https://gitlab.com/gitlab-org/gitlab/-/blob/75dfa40c9ff9e0aa37db5c1754c816946f9bc81d/ee/lib/search/zoekt/code_query_builder.rb#L33-38 — project search filters by repository id, group search by `traversal_ids`. </details> ### Related - https://gitlab.com/gitlab-org/gitlab/-/work_items/630027 — repositories `ready` in the database with no shard on disk. Seen as a follow-on here: after a same-root project move, a later reindex wrote a new shard, and the indexer's deferred delete from the move then removed it. Could be https://gitlab.com/gitlab-com/request-for-help/-/work_items/5373