Zoekt: Add a worker to create indexing tasks

Background

This is continuation of Zoekt: Add Search::Zoekt.index! method (#442886 - closed)

Proposal

In Zoekt: Refactor the worker into index_async/ind... (!148827 - merged) we've added ::Search::Zoekt.index_async and ::Search::Zoekt.index_in.

Now we need to add a new worker that's going to create zoekt_tasks behind a feature flag and hook it into the same methods.

The logic of this new worker Search::Zoekt::IndexerWorker could be something like this draft:

Click to expand

REINDEXING_CHANCE_PERCENTAGE = 0.5

def perform(project_id, options = {})
  return unless ::Feature.enabled?(:index_code_with_zoekt)
  return unless ::License.feature_available?(:zoekt_code_search)
  return if Feature.disabled?(:zoekt_tasks_api)

  project = Project.find_by_id(project_id)
  return false unless project
  return unless project.use_zoekt?
  return unless project.repository_exists?
  return if project.empty_repo?
  
  Search::Zoekt::Index.for_root_namespace_id(fetch_root_namespace_id(project)).find_each do |index|
    repo = index.zoekt_repositories.find_or_create_by(project_id: project.id)

    task_type = :index_repo
    task_type = :force_index_repo if !!options['force'] || random_force_reindexing?

    repo.tasks.create!(task_type: task_type)
  end
end

private

def random_force_reindexing?
  return false if Feature.disabled?(:zoekt_random_force_reindexing, type: :ops)

  rand * 100 <= REINDEXING_CHANCE_PERCENTAGE
end