Skip to content

Add gitlab-zoekt-indexer image

Dmitry Gruzd requested to merge add-gitlab-zoekt-indexer into master

What does this MR do?

This MR adds a new image for https://gitlab.com/gitlab-org/search-team/gitlab-zoekt-indexer, which is going to replace zoekt-dynamic-indexerver. The latter has been introduced in !1320 (merged)

How to test it (docker)

  1. docker build -t gitlab-zoekt-indexer -f gitlab-zoekt-indexer/Dockerfile gitlab-zoekt-indexer
  2. docker build -t gitlab-zoekt-webserver -f gitlab-zoekt-webserver/Dockerfile gitlab-zoekt-webserver
  3. docker volume create zoekt-index
  4. Uncomment listen_addr = "127.0.0.1:2305" in $GDK_DIR/gitaly/praefect.config.toml
  5. Execute gdk restart praefect
  6. docker run -p 6060:6060 --name gitlab-zoekt-indexer --rm -v zoekt-index:/data/index gitlab-zoekt-indexer
  7. docker run -p 6070:6070 --name gitlab-zoekt-webserver --rm -v zoekt-index:/data/index gitlab-zoekt-webserver
  8. Get JSON payload by running these commands in rails c 
project = Project.find_by_full_path('flightjs/Flight')
payload = {
        GitalyConnectionInfo: {
          Address: 'tcp://host.docker.internal:2305',
          Storage: project.repository_storage,
          Path: "#{project.repository.disk_path}.git"
        },
        RepoId: 7,
        FileSizeLimit: 2097152,
        Timeout: "10m"
  }.to_json
  1. Execute curl -XPOST -d '<PAYLOAD>' -H 'Content-Type: application/json' http://127.0.0.1:6065/indexer/index
  2. See search results
curl -XPOST -d '{"Q":"test","RepoIDs":[7]}' 'http://127.0.0.1:6070/api/search' | jq .

How to test it (docker-compose)

  1. docker build -t registry.gitlab.com/gitlab-org/build/cng/gitlab-zoekt-indexer:master -f gitlab-zoekt-indexer/Dockerfile gitlab-zoekt-indexer
  2. docker build -t registry.gitlab.com/gitlab-org/build/cng/gitlab-zoekt-webserver:master -f gitlab-zoekt-webserver/Dockerfile gitlab-zoekt-webserver
  3. Uncomment listen_addr = "127.0.0.1:2305" in $GDK_DIR/gitaly/praefect.config.toml
  4. Execute gdk restart praefect
  5. docker-compose up zoekt_indexer zoekt_webserver
  6. Get JSON payload by running these commands in rails c 
project = Project.find_by_full_path('flightjs/Flight')
payload = {
        GitalyConnectionInfo: {
          Address: 'tcp://host.docker.internal:2305',
          Storage: project.repository_storage,
          Path: "#{project.repository.disk_path}.git"
        },
        RepoId: 7,
        FileSizeLimit: 2097152,
        Timeout: "10m"
  }.to_json
  1. Execute curl -XPOST -d '<PAYLOAD>' -H 'Content-Type: application/json' http://127.0.0.1:6065/indexer/index
  2. See search results
curl -XPOST -d '{"Q":"test","RepoIDs":[7]}' 'http://127.0.0.1:6070/api/search' | jq .

Related issues

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com
  • When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Integration tests added to GitLab QA
    • GitLab QA tests for the Zoekt integration would need to be added after it is supported in Omnibus. It's not a near term goal to support this for self-managed so prioritizing these containers to support deployment on GitLab.com makes more sense.
  • The impact any change in container size has should be evaluated 
    $ docker image ls | fgrep indexer
gitlab-zoekt-indexer                                              latest               b1a5bfeb06cc   33 seconds ago   212MB
Edited by Dmitry Gruzd

Merge request reports