Search: read from OpenSearch and retire the Logstash pipeline

The artifact search stops reading Elasticsearch and starts reading OpenSearch, and the nightly Logstash reindex is deleted.

The index is rebuilt once a night by a Logstash pipeline on a 50 4 * * * schedule, so an approved edit can take a day to appear. The replacement is built and sits below this MR in the stack: writes enqueue into reindex_outbox, reindex_poller drains it and rebuilds each document with ArtifactDocumentBuilder, and search_backfill does the same in bulk. This MR is the last step, pointing the read path at the new index and removing the old pipeline.

Merge order matters, because none of this has reached phoenix/develop yet. This MR targets phoenix/gsoc/search/hardening and the stack below it is still open, so the pipeline and this cutover reach develop together, in order. Flipping the default ahead of the branches that write the new index would point search at an index nothing fills.

The merge is the cutover. Nothing deploys piecemeal: the flip takes effect only when the stack lands, and from then search reads OpenSearch and nothing writes Elasticsearch again. The first half of this branch added per-request engine selection so the two could be compared; the second flips the default and deletes the pipeline.

Behaviour changes

1. X-Search-Engine response header, set by both search controllers before the engine is called, so it survives the admin viewer's rendered error states.
2. The admin document viewer renders an outage state instead of an uncaught transport exception, naming the address it tried.
3. The admin document viewer renders a miss state instead of an uncaught exception.
4. An unknown admin page name is a 404, not a 500. The old behaviour told an operator the id was not in the index when no id had been looked up.
5. The index-time badge renders only when the document carries a timestamp. The template read $result['@timestamp'] unguarded; the builder never emits that field, and DateTime::parse(null) returns the current time rather than failing, so every result on an OpenSearch page would have read "indexed just now" permanently. Cost: administrators and editors lose the badge on default-engine pages, and with it the only in-app link to the admin _doc viewer, which stays reachable by URL.
6. The default engine is OpenSearch. ?engine=elasticsearch selects Elasticsearch explicitly; every other value, including absence and an unrecognised name, serves OpenSearch.

Evidence

226 request bodies generated from ElasticSearchQuery itself, replayed against both engines by base URL on _search and _count. It covers transliteration search in every mode, the field and value surface, where() filters, keyset pagination, access-control variants, all 18 getSortFields() entries, all 27 aggregations the page requests, and translation search. 0 engine errors on either engine, 147 of 226 bodies with hits on both.

• All 18 sort fields: ordered ids and totals identical.
• Production-shaped sort (_score desc plus id asc, what every shipped request sends): 74 of 76 bodies ordered-identical, 71 of 76 identical on totals.
• 18 of 27 aggregations identical bucket for bucket, count for count, in order.
• Tie-class probe: 56 bare-_score bodies probed, membership identical on all 56. The 51 that differ in within-window position are BM25 scores differing in the fifth decimal between the two Lucene versions, and no shipped request is affected because all of them append the id tiebreaker.

Every difference lands in one of these rows. There were no others.

difference	measured	disposition
Long documents miss whole-value transliteration search	217 documents exceed the keyword bound on atf_transliteration, 122 on the surface container, 0 on lines	mapping; residue enumerated, unavoidable for 26 that exceed Lucene's own term limit
The umun2 mode-full gap	ES 139 against OS 134, gap exactly {469516, 469530, 469683, 469698, 480190}; line and section modes id-identical at 139 and 111	same residue
Larger mode-full count gaps	udu %OR% gu4 29,445 / 29,261; udu %NOT% UMUN2 19,470 / 19,321. Subtracting the 217-id residue makes every one exact	same residue
Publications the legacy pipeline dropped	574 artifacts gain a non-empty publication	no action, builder correct
Orphaned-FK publication data	the simple:publication "Or" set is 1,718 wider and partitions exactly: 527 all-NULL-type, 1,184 dangling-FK, 7 mixed	no action, sign-off already open
Duplicated update events	artifact 3510 has 9 entries with 8 distinct against 8	no action, builder correct
Two documents the legacy index is stale about	724 and 1023 carry a publication with no row in the database	no action, builder matches the database
Empty string against null under exists	3 artifacts enter the asset_license_id "no value" bucket on OpenSearch only	no action
Five facets lose their "no value" option	f[<facet>][]=no value answers 0 on OpenSearch. Of the other 22 facets, 19 are equal and 3 differ without emptying, all three covered above	blocking comment below; the fix is its own MR
Score ties order differently	51 of 56 bare-_score bodies, id sets identical	no action, engine
Capped totals	hits.total caps at 10,000 on both; the comparison unit is _count and id sets	no action, engine

The publication and update rows are divergence classes the builder's full-index parity sweep already enumerated; they are cited, not re-derived. The "" against null axis is one the document-level comparator treats as equal by design, so it is invisible there and visible at the query layer. It is the only difference in the run with a visible effect on the search page.

Four preconditions were checked independently of parity, because parity compares totals, id sets, buckets and order and cannot see a render-layer read or a missing service: the compose matrix read off the files; the escape hatch proved live over HTTP by each engine's _stats/search counter on all four parameter cases, on the page and the JSON route; no row left predicted; and the badge checked by rendering both arms with an administrator session, where the Elasticsearch page shows 25 badges and 25 viewer links and the OpenSearch page shows neither and renders whole. The template's read surface was swept: 32 distinct $result['...'] names against the mapping's 82 declared leaves, 4 undeclared, of which @timestamp is now guarded and the licence triplet is ?? null guarded and carried by no document on either index.

Before this deploys anywhere

variant	elasticsearch	opensearch	reindex_poller
dev	yes	yes	yes
gea	yes	no	no
mpi	yes	no	no
alliance	no	no	no

gea and mpi must not take this merge before an OpenSearch service and a poller exist there. The result is an outage rather than a stale index. The flipped default resolves ServiceUrls.opensearch, whose hostname does not exist on those networks, so every search request 500s from the moment this deploys. Adding those service blocks needs host paths this project cannot see, so it is asked as a comment rather than guessed at.

The runbook has the deployment order and the rebuild path. In short: OpenSearch reachable, then search_index_create, search_index_swap --alias both, search_backfill (idempotent, about two and a half minutes at 419,480 documents), then the poller, and only then this merge.

Not in this MR

• The "no value" facet fix. One aggregation redesign across all eighteen terms facets, moving facet counts on both engines. Merging waits on the blocking comment below, since the answer decides whether it lands first, in which case the facet figures above need re-measuring.
• The Elasticsearch service, Kibana, its volumes and the ?engine=elasticsearch arm. The arm stays as the comparison instrument for the review window; without an Elasticsearch service it is a 500, the same uncaught-transport contract the OpenSearch arm shipped with before the flip. A follow-up retires all of it once the window closes.
• A real indexed-at field, which would make the badge true again but owns a design question this MR should not answer: what a build-time timestamp does to the _source equals build() invariant reindex_reconcile gates on.
• A test holding the template's $result reads against the mapping. The sweep above was run by hand.

Pre-existing and not introduced here: the licence triplet reads, carried by no document on either index; the publication and update divergence classes awaiting sign-off; and README.md pointing at a dev/Search_Setup.md that does not exist.

dev/docker-compose.yml and dev/config.json are generated locally, so regenerate them after pulling. A local dev/config.json still carrying a logstash key with "is_default": true makes cdli.py ... up fail with no such service: logstash; drop the key.