Search: proveniences indexed, captured, and served from the engine

Proveniences are searched today with a raw LIKE against MariaDB, and none of the search-engine machinery the artifact pipeline gained applies to them. This MR is the first non-artifact entity to go through that pipeline end to end: an index built from PHP, kept fresh by the outbox and poller, serving the /proveniences page. The database stays as a live fallback. It is the template the remaining entities copy.

The document and its index. ProvenienceDocumentBuilder emits seven flat fields per row; ProvenienceIndexMapping declares them under the same conventions as the artifacts mapping, text plus a bounded keyword, no analyzers. provenience_ascii strips combining marks with the same helper the artifact document uses. Registration is one line in each of the four maps plus the DI container, and a single test holds all five together. Proveniences hard-delete, so a backfill cannot remove an already-deleted document: the clean rebuild for this entity is a fresh _vN plus an alias swap. The orphan and its removal were walked on the running stack (second comment).

One artifact-side correctness fix arrives with the first non-artifact entity, and it had to: ReindexReconcileCommand counted outbox coverage on id alone, and ids are unique per entity type, not across them, so a provenience row could suppress an artifact's repair. Both subqueries now carry entity_type = 'artifact'. Reconcile itself stays artifact-only.

Capture. Approving a provenience change writes a direct outbox row carrying the id, so a delete needs no lookup after the row is gone and the edited entity is rebuilt ahead of any cascade. Region and location edits fan out one outbox row per affected document type, from a single declared map (CASCADE_TARGET_TYPES), with resolver arms mirroring the builder's joins. A tokenizer test fails the moment a builder reads a table with no capture declaration, because that is the one failure here nothing downstream can see. Artifact behavior is unchanged throughout. Two artifact tests are touched: RegionsControllerTest gains the second row a region edit now writes, artifact row first and its payload untouched, and IndexMappingRegistryTest hands its entity-list pin to SearchRegistrationTest, which holds every registration map to one list.

The read side. /proveniences now asks OpenSearch for the matching ids and the region facet in one _search, then lets MySQL do what it already did: hydrate, order, paginate, export. Sorting and pagination are byte-identical because the same code produces them; what changed hands is the matching. Parity with LIKE '%value%' under utf8mb4_unicode_ci is reached by a wildcard query on provenience_ascii.keyword with case_insensitive: true, the query value folded by the same helper that folded the indexed text. A shared LikeWildcard class mirrors the database escape pipeline symbolically, with all six backslash corners measured against the real finder and pinned as unit tests. The sorted-page-from-the-engine shape was measured and dropped for this entity: no engine sort reproduces the collation's order (479 of 738 positions differ against the closest precomputed key), and on 738 rows it buys nothing. The rule is written down; Publications, at 16,686 rows, plans the sorted-page shape from the start.

Three measured divergence classes are pinned, thirteen pins in all, asserted in both directions by the parity harness (265 parameter sets derived from the live data, built as a trait each remaining entity plugs into). Two classes are the engine matching less than MySQL. The third is the engine matching more: seven names stored with combining marks that today's site search cannot fold, a pre-existing data quirk the engine corrects. The detail is in the first comment.

Failure handling splits deliberately. If the engine is unreachable, the controller logs one warning and serves the exact query the page used before, old dropdown included. A refused query surfaces as an error instead, because an alias nobody created or a malformed body is a bug in this code, and plausible rows from the database would leave it running unnoticed. Both were exercised against the running app.

One behavior changes on purpose. The region dropdown becomes a facet: the regions in the current result set, with counts, from the same query that returned the ids, so a count cannot disagree with the rows beside it. The fallback path renders the old dropdown unchanged. The autocomplete endpoint stays on the database deliberately.

Not in this MR

No artifact read-path change: artifact search still queries Elasticsearch, and engine selection is the cutover MR. No diacritic folding for atomic letters, no non-ASCII case folding, no Unicode normalization on either side (all issue #2018's analyzer version), and no change to the seven combining-mark rows (first comment). All are pinned until then. No resume, checkpoint or progress UI for the backfill. view, history, add, edit and every template are unchanged.

Runbook

Create and swap before the first read: search_index_create --entity provenience, then search_index_swap --entity provenience --version 1 --alias both, then search_backfill --entity provenience. A search against an alias nobody created raises instead of returning nothing, which fixes the deployment order. The backfill refuses under the container's 128M default and names the invocation to use. The engine path reflects an approved edit after one poller cycle; a row deleted whose document has not caught up drops out at hydration and self-heals in the same window.

Evidence, timings and the testing map are in the second comment.

Edited by sung

Merge request reports

Loading
Loading