Converting FFI to HTTP for Full Indexing

Omar Qunsulrequested to merge
105-full-indexing-using-http into main

What does this MR do and why?

Converting FFI to HTTP for Full Indexing

Addressing: #105 (closed)

This also removes previous functionality of including Kuzu (Knowledge Graph Indexing and Querying) into Zoekt-Indexer in favor of the new service architecture. See these two links for more context:

How to set up and validate locally

A. When NOT running in Knowledge Graph Mode (this should not happen on Production)

Note: This should not happen in Production, because when enabling the Knowledge Graph Mode, the node should modify its state on the Monolith to be running in the right mode. That's to be implemented in a later issue.

  • Checkout this repo and run make build-unified
  • Enable the FF knowledge_graph_indexing on some project locally in GDK
  • Enable Knowledge Graph service on all the Zoekt Nodes via Rails Console Search::Zoekt::Node.update_all(services: [0, 1])
  • In GDK stop all the zoekt-indexer instances: gitlab-zoekt-indexer-development-1 and gitlab-zoekt-indexer-development-2. Run the process manually in a shell ./gitlab-zoekt-indexer/bin/gitlab-zoekt indexer -index_dir zoekt-data/development/index -listen :6080 -secret_path /Users/omar/gdk/gitlab-shell/.gitlab_shell_secret -self_url "http://localhost:6080" -search_url "http://localhost:6090" -gitlab_url http://gdk.test:3000. Make sure to change to change the paths.
  • (ALTERNATIVE) Keep gitlab-zoekt-indexer-development-1 running after you restart it. And run gdk tail gitlab-zoekt-indexer-development-1 to see the output.
  • Push some new code to the main branch of the project that you enabled the FF. You should see something like this in the output
{"time":"2025-09-11T15:25:07.002351+02:00","level":"ERROR","msg":"received graph indexing task. But not running in knowledge Graph Mode","namespaceId":102,"task":"index_graph"}

B. When running in Knowledge Graph Mode

  • In a separate tab, run go run gitlab.com/OmarQunsulGitlab/GoEcho@latest unix:///tmp/echo.sock
  • Follow the same steps as before. But you need to set these environment variables
export GITLAB_ZOEKT_GKG_INDEX_URL=unix:///tmp/echo.sock
export GITLAB_ZOEKT_GKG_QUERY_URL=unix:///tmp/echo.sock

In the output of the Echo Server you should see

Received request:
User-Agent: Go-http-client/1.1
Content-Length: 142
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip
{"repositoryPath":"/var/folders/j1/th4dzm2n3hd1lqv4dx9lpkmr0000gn/T/gitaly-repo-archive-2146066722","namespaceId":102,"repoId":19,"threads":0}

In the output of the Zoekt Indexer you should see:

{"time":"2025-09-11T15:24:42.551371+02:00","level":"INFO","msg":"Knowledge Graph Service: Starting Full Indexing","NamespaceID":102}
{"time":"2025-09-11T15:24:42.551587+02:00","level":"INFO","msg":"Knowledge Graph Service: Finished Full Indexing","NamespaceID":102,"StatusCode":200}

Closes #105 (closed)

Edited by Omar Qunsul

Merge request reports