Zoekt: Add Search::Zoekt.index! method

Background

This issue is to add Search::Zoekt.index! method so that we can replace all calls to index a repository (Zoekt::IndexerWorker.perform_async) with it.

Proposal

The logic should be something like this draft:

Click to expand

REINDEXING_CHANCE_PERCENTAGE = 0.5

def index!(project, **options)
  return unless ::Feature.enabled?(:index_code_with_zoekt)
  return unless ::License.feature_available?(:zoekt_code_search)
  return unless project.use_zoekt?
  return unless project.repository_exists?
  return if project.empty_repo?

  Zoekt::IndexerWorker.perform_async(project.id, options)

  create_index_tasks
end

private

def create_index_tasks
  return if Feature.disabled?(:zoekt_tasks_api)

  Search::Zoekt::Index.for_root_namespace_id(fetch_root_namespace_id(project)).find_each do |index|
    repo = index.zoekt_repositories.find_or_create_by(project_id: project.id)

    task_type = :index_repo
    task_type = :force_index_repo if !!options['force'] || random_force_reindexing?

    repo.tasks.create!(task_type: task_type)
  end
end

def random_force_reindexing?
  return false if Feature.disabled?(:zoekt_random_force_reindexing, type: :ops)

  rand * 100 <= REINDEXING_CHANCE_PERCENTAGE
end

As part of the MR we should also replace Zoekt::IndexerWorker.perform_async calls with the new method.

Another potential option is to create a new worker Search::Zoekt::IndexerWorker that we'd call from index! since we have quite a bit of logic and DB queries in it. That way we can replace the legacy worker with the new one.