Search: reindex poller ops and reconciliation (health, cleanup, reconcile, service)

Closes #2610

Ops layer for the reindex poller: three commands, plus a compose service to run it. Also moves one lock check that the restart policy in this MR makes necessary.

reindex_poller_health checks the outbox and exits non-zero if a pending row has waited past the threshold (default 5 min, --stale-minutes) or a processing row is stuck past the watchdog window. It prints the failed count too, but that doesn't change the exit. Doubles as the container healthcheck. Note: a big but legitimate backlog also shows unhealthy while it drains, so "Up (unhealthy)" means behind, not down.

reindex_maintenance deletes done rows older than --keep-days (default 7). failed rows stay (requeue is manual). Retention has to stay above the reconcile window (25h) so the sweep can still see the rows.

reindex_reconcile is the backstop behind the capture sites, in two modes. Both enqueue into the same outbox, so repairs run through the normal poller pipeline at cascade priority.

  • sweep (default) finds changes since --since (default 25h) with no covering outbox row and enqueues them. Only artifact_assets and authors have a timestamp to scan by, so it's limited to those two. Coverage is by row status: pending/processing/failed cover a change (they rebuild from current state), done only if it built after the change. So a re-run is a no-op, and a stale rebuild can't hide a later change.
  • drift check (--sample N, or --ids for a set) rebuilds the artifacts and diffs them against the index via _mget, key-sorted so only real drift shows, and enqueues repairs. Catches what the sweep can't (the timestampless tables, deletes). --dry-run works on both.

The container is a reindex_poller compose service, dev only for now (enabled: false; enable it together with opensearch or the compose won't parse). It runs on the Dockerfile_composer image, not the web one, because the web image's CLI can't boot CakePHP at all (no intl, no pdo_mysql). memory_limit=512M is set on the command (the 128M default is too low). restart: unless-stopped handles crashes and lock loss. One gotcha from testing: docker kill counts as a manual stop so it won't restart, but a real crash or a killed DB connection does. gea/mpi don't have opensearch yet, so the service ships there later with the OpenSearch setup work.

Poller change: the lock re-check moved above the busy-path continue, so a poller that lost its lock while still finding work exits instead of claiming again. It couldn't happen before, but the restart policy makes a second instance possible now. Also rewords the verbose line to "claimed" and adds a test seam. Nothing else in the poller, capture side, builder, indexer, resolver, or claim changes.

Scheduling

No scheduler in the app, run these from host cron, exec'd into the container:

30 4 * * *  docker exec reindex_poller php /srv/app/cake/bin/cake.php reindex_maintenance
40 4 * * *  docker exec reindex_poller php /srv/app/cake/bin/cake.php reindex_reconcile
50 4 * * 0  docker exec reindex_poller php -d memory_limit=512M /srv/app/cake/bin/cake.php reindex_reconcile --sample 1000

Testing

Unit tests for the three commands, plus an end-to-end test that seeds drift and checks it's detected, repaired, and converges after a poll cycle.

Merge request reports

Loading