Skip the WAL health check when indexing on Zoekt search fallback

What does this MR do and why?

When a project code search falls back from Zoekt because the project has no Zoekt repository yet, zoekt_index_project_on_search_fallback queues indexing for it. Today that job is Search::Zoekt::IndexingTaskWorker, which is deferred whenever any database health indicator signals stop: by 10 minutes, then 20, then up to 30 minutes for each later deferral (incremental_database_health_defer_delay, enabled on GitLab.com). On GitLab.com the WriteAheadLog indicator signals stop for long stretches on most weekdays (gitlab-com/gl-infra/production-engineering#30227), so the job a searcher is waiting on waits with all other indexing.

This MR adds Search::Zoekt::FallbackIndexingTaskWorker. It runs the same IndexingTaskService index_repo path but does not check WriteAheadLog. Autovacuum on zoekt_repositories and zoekt_tasks, PatroniApdex and WalRate still apply. The search fallback uses it behind the new zoekt_search_fallback_skip_wal_check flag (gitlab_com_derisk, default off).

Only indexing that a user is waiting on skips the check. Push-triggered, bulk and forced indexing keep using IndexingTaskWorker with every indicator.

Why a separate worker, and why only the search fallback
  • defer_on_database_health_signal sets its indicators per worker class. The block form can change only the schema and tables per job, so the only way to check different indicators for these jobs is a different worker class.
  • A deferred job is re-enqueued and doesn't re-check until its delay is up. It also no longer holds its deduplication lock, so a fallback job is not deduplicated against a deferred push job. Today the fallback job is deferred by the same WriteAheadLog signal, and only runs if it happens to check while the signal is clear.
  • New projects are indexed on push. The search fallback catches searches that arrive before that indexing has run. Making only the fallback skip the check keeps the bypass to the projects someone is searching, instead of every new project, and leaves the rest of indexing to wait out the backlog.
  • concurrency_limit is counted per worker class, so the new worker gets its own 100 slots. It only writes the project's indexing task rows, one per replica, and duplicate jobs for a project are merged, so the extra pool should stay nearly empty.
  • If a fallback job and a push job for a project run close together, they can write two tasks when one of them is sampled into a forced reindex (0.25% per job by default); the cost is one extra indexing.
  • Without a Zoekt repository, project code search falls back to basic search, so the searcher still gets results for that first search.

References

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

  1. In the Rails console, check the worker's indicators:

    Search::Zoekt::FallbackIndexingTaskWorker.database_health_check_attrs[:indicators]
    # => [AutovacuumActiveOnTable, PatroniApdex, WalRate], no WriteAheadLog
  2. Enable the flags:

    Feature.enable(:zoekt_index_project_on_search_fallback)
    Feature.enable(:zoekt_search_fallback_skip_wal_check)
  3. With Zoekt enabled for a group, delete a non-empty project's Zoekt repository (Search::Zoekt::Repository.for_project_id(project.id).delete_all) and run a code search in that project.

  4. log/sidekiq.log shows Search::Zoekt::FallbackIndexingTaskWorker for the project, and the project has a Zoekt repository again.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Ravi Kumar

Merge request reports

Loading
Loading