Pull tasks from the tasks API

What does this MR do and why?

This MR processes the response from the Tasks API. If the task is index, it will perform indexing; if the task is delete, it will perform the deletion. Both processes will happen asynch.

How to set up and validate locally

• Enable Zoekt
• Create two new fresh projects in the enabled zoekt namespace. Add at least one commit to each project.
• Apply the following changes in this file https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/admin/search/zoekt.rb

Choose one project as index_project and the other as delete_project from the newly created two projects. Replace the value of Storage, Path, and RepoId with your values.

--- a/ee/lib/api/internal/search/zoekt.rb
+++ b/ee/lib/api/internal/search/zoekt.rb
@@ -34,7 +34,33 @@ def logger
 
                   # We don't want to register (save) the node if the feature flag is disabled
                   if Feature.disabled?(:zoekt_internal_api_register_nodes, type: :ops) || node.save
-                    { id: node.id }
+                    {
+                      id: node.id,
+                      tasks: [
+                        {
+                          name: 'index',
+                          payload: {
+                            GitalyConnectionInfo: {
+                              Address: "unix:#{ENV["GDK_DIR"]}/praefect.socket",
+                              Token: nil,
+                              Storage: index_project.repository_storage,
+                              Path: "#{index_project.repository.disk_path}.git"
+                            },
+                            Callback: { name: 'index', payload: { foo: 'bar' } },
+                            RepoId: project.id,
+                            FileSizeLimit: Gitlab::CurrentSettings.elasticsearch_indexed_file_size_limit_kb.kilobytes,
+                            Timeout: '1800s'
+                          }
+                        },
+                        {
+                          name: 'delete',
+                          payload: {
+                            RepoId: delete_project.id
+                          }
+                        }
+                      ]
+                    }
                   else
                     unprocessable_entity!
                   end

• Switch the Zoekt version

gdk config set zoekt.version 424123-implement-pull-tasks && gdk reconfigure

• Run zoekt indexer in GDK mode
• tail the zoekt.log. And verify that you get the callback logs for delete and index. Following is an example

{"severity":"INFO","time":"2024-02-27T20:34:36.734Z","correlation_id":"01HQP4DGHQ2Y1SRDTBT8D8K96G","meta.caller_id":"POST /api/:version/internal/search/zoekt/:uuid/callback","meta.remote_ip":"127.0.0.1","meta.feature_category":"global_search","meta.client_id":"ip/127.0.0.1","class":"API::Internal::Search::Zoekt","node_id":1,"callback_name":"delete","payload":{},"additional_payload":{},"success":true,"error_message":null}
{"severity":"INFO","time":"2024-02-27T20:34:36.815Z","correlation_id":"01HQP4DGM945KVAMRHA87PN1FF","meta.caller_id":"POST /api/:version/internal/search/zoekt/:uuid/callback","meta.remote_ip":"127.0.0.1","meta.feature_category":"global_search","meta.client_id":"ip/127.0.0.1","class":"API::Internal::Search::Zoekt","node_id":1,"callback_name":"index","payload":{"foo":"bar"},"additional_payload":{"repo_stats":{"index_file_count":1,"size_in_bytes":42008}},"success":true,"error_message":null}

Related: gitlab#424123 (closed)