Search: rebuild an entity's whole index from bin/cake

The previous MR made it possible to create artifacts_v1 from PHP. Nothing fills it. The poller only writes what the outbox asks for. This MR adds the command that walks a source table and rebuilds every document in it.

bin/cake search_backfill --entity artifact. Keyset pagination over artifacts, the entity's builder over each page, one _bulk per page. Re-running rewrites the same documents, so it is also the recovery step for a run that stopped part way.

No --index option. The write target is whatever artifacts_write points at.

I planned an override and dropped it. The poller stays live during a backfill and writes through the write alias. An explicit target that no alias points at would take every edit approved during the run into the old index. The backfill would still report success. The loss would surface only after the read swap, and reindex_reconcile reads the write alias too, so it would audit the old index and call it healthy.

Both legitimate uses are already covered. Filling a new version is search_index_create, search_index_swap --alias write, this command, then --alias read. Filling the live index is this command with no swap.

The poller keeps running, and one window needs closing. Both writers address the same index, so most orderings are safe on their own. The unsafe one is an edit the poller writes between a batch's build and its push. The backfill overwrites that fresh document with the one it built moments earlier. The outbox row is done by then, so nothing re-drives it. A retirement racing an index action does the same thing in reverse and resurrects a retired artifact.

The window is under a second, and CDLI approves a handful of edits a day, so a real rebuild would rarely hit it. But the loss is permanent and sampling cannot find it. A 1,000-id sample over 419,480 documents draws a given clobbered id about one time in 400.

So the run ends by re-enqueuing the ids the poller was working on, and the poller rebuilds them afterwards.

SELECT entity_ids FROM reindex_outbox
 WHERE entity_type = 'artifact'
   AND (processed_at IS NULL OR processed_at >= <run start>)

processed_at and not created_at is the part worth a second look. enqueue() runs inside the approving transaction, so created_at is stamped at the INSERT while the row becomes visible only at commit. A transaction that inserts before the run starts and commits after it is fully concurrent with the backfill, and a created_at window would miss it. A tolerance does not help, because it would have to exceed the longest open write transaction. Every write of processed_at is NOW() on the database clock, after the push it reports.

The predicate is a superset on purpose. The IS NULL arm covers the row whose bulk landed before the last push but whose done-marking had not run. It also sweeps backlog the poller never started, which costs a rebuild it would have done anyway.

Failures are per document, and only sometimes per run. BulkResult reports per id, so a rejected document is reported and the scan carries on. A dead engine aborts the run instead, because continuing would print the same transport error.

The two are told apart by the reason the indexer attributed, not by counting failed ids. The final batch is 126 ids and smaller entities have smaller ones, so a couple of genuine mapping rejections can fail every id in a batch. So a batch aborts only when nothing in it succeeded and every failure is request-level. Mixed batches are real. The split recurses with the same accumulators, so one over-limit document can fail beside multiple successes.

An abort skips the tail. The message says so, and says that re-running is what re-protects the window.

Two preflights, before anything is written. The write alias has to resolve to exactly one index. And memory_limit has to be at least 512M, because the container default is 128M and a batch of 500 ATF-heavy artifacts peaks at 166MB. The refusal names the invocation to use instead of calling ini_set() for the operator. The shorthand needs parsing rather than casting, since 524288K is 512M to the byte.

No --verify. reindex_reconcile --sample N and --ids already build fresh, _mget the index and classify, and they read the write alias. During a rebuild that is the index the backfill just filled. A second comparator here would duplicate normalize() and classify(), and two spot-check paths that disagree is a bug the gate cannot resolve.

Measured

Full rebuild on the local stack, poller live throughout.

• 430,626 ids scanned, 419,480 indexed, 11,146 deleted, 0 failed, in 2m 18s.
• GET artifacts_write/_count returns 419,480, matching the corpus exactly.
• Peak memory 132MB, past the container's 128M default. The run would have fataled without php -d memory_limit=1G.
• reindex_reconcile --sample 1000 gave 981 match and 19 consistent_delete, nothing else. A hand-picked five came back the same.
• Five documents were also diffed straight from _source against a fresh build(), outside the reconcile path. All identical.
• An edit approved mid-run was caught by the tail and rebuilt correctly after one poller cycle.
• A re-run left _count unchanged and moved _version from 1 to 2, so documents were rewritten rather than skipped.

Runbook

Do not run search_index_swap while a backfill is in flight. The alias resolves per request and the preflight runs once, so a mid-run swap splits the run across two indexes.

Abandoning a rebuild is not free. Swap the write alias back, then re-enqueue the ids written only into the abandoned index. That is the tail query above with the write-swap time as the start, fed to reindex_reconcile --ids.

Two smaller things. The tail only protects entities that have a capture path. And the 512M floor is derived from ATF-heavy artifacts but applies to every entity, so it may be worth loosening per entity later.

Not in this MR

No resume, no checkpoint, no reindex_jobs table, no admin progress UI, no poller pause, no --engine=both.

Nothing existing changes behaviour. One production file outside the new ones is edited. IndexLifecycleService::holdersOf() goes from private to public so the preflight can ask it.

One limit worth stating. A backfill cannot remove a document whose database row is gone, because the scan only yields ids that exist as rows. Artifacts soft-delete, so retired rows still produce deletes and the case is rare here. It matters for any later entity that hard-deletes. Either it has a soft-delete column to scan, or its only clean rebuild is into a fresh version.

Testing

BackfillServiceTest drives a scripted id space through the real batching, builder call and push, with the scan behind its seam. BackfillServiceScanTest covers that seam against the real table. SearchBackfillCommandTest runs the command against both factory seams with a real outbox behind it, including the tail window seeded across the processed_at and status combinations. SearchBackfillE2ETest runs the real thing against local OpenSearch and cdli_db, bounded to the first 120 ids, which is the shortest range holding a retired artifact.

Note:

The last commit is unrelated to the backfill. Three older E2E tests deleted their scratch index in tearDown() even when setUp() had refused to run, so the guard protecting a standing artifacts_v1 was what destroyed it. A $guardPassed flag fixes all four.