Zoekt Search - Rocky Mongare

module-name: "Zoekt Search"
area: "Global Search"
gitlab-group: "Global:Search"
maintainers:
  - bnyaringita
  - cleveland
  - aysegula

Overview

Goal: Learn the basics of Exact code search and how to setup Zoekt.

Length: 2-4 hours

Objectives: At the end of this module, you should be able to:

  • Learn the background of Zoekt
  • Familiarize with the installation methods
  • Setup a simple Zoekt instance
  • Learn troubleshooting techniques

Stage 0: Create and commit to the module

  1. Create an issue using this template by making the Issue Title: Zoekt Search - <your name>

  2. Add yourself and your trainer as the assignees.

  3. Commit to this by notifying the current experts that they can start routing Zoekt Search questions to you.

  4. Optional: Set a milestone, if applicable, and a due date to help motivate yourself!

  5. Update your Support Team yaml file to indicate that you've started learning this knowledge area:

    knowledge_areas:
- name: Zoekt Search
  level: 1

Consider using the Time Tracking functionality so that the estimated length for the module can be refined.

Stage 1: Introduction to Zoekt

Zoekt(/zukt/) is an open-source search engine designed specifically to search for code. Zoekt was developed to address limitations with code search in Elasticsearch. Zoekt is enabled in paid subscriptions for GitLab.com. These setup steps are for Self-Managed instances only. Zoekt is not yet enabled for GitLab Dedicated.

Note: If you are already familiar with Zoekt, skip to Stage 2: Technical Setup.

  1. Read the motivation behind moving Zoekt implementation for exact code search:

    1. Why Zoekt and the problems it would solve. You can also read more on this epic.
    2. Design and implementation details. Browse through the architecture, indexing flow and search flow diagrams to get a rough idea of the backend implementation.
    3. (Optional) Read about the origin of Zoekt's development and how Sourcegraph makes use of Zoekt

  2. Familiarize with user documentation.

    1. Exact code search overview.
    2. Available installation methods.
    3. Troubleshooting Zoekt

  3. Familiarize where to get help:

    1. Known issues:
      1. GitLab Zoekt Indexer
      2. Global Search group issues
    2. Ask for help in the support pod - #spt_pod_advanced-search. This channel will also get the attention of the Search engineering team.
    3. To ask about features, clarifications or fast-track an issue you can ask in the dev channels - #g_global_search or the wider #search channel.
    4. Create a Request For Help issue

Stage 2: Technical Setup

You can choose to run Zoekt in the same server as GitLab or on a separate server. To run on a separate server, ensure you have changed the Gitaly listening interface.. For the purposes of testing Zoekt in this module, you can disable Elasticsearch if previously setup and only use Zoekt for code search.

Option 1A: Using the Zoekt Helm chart with GitLab Helm Chart

  1. Prerequisites: Install GitLab using Helm chart. You can use the GitLab Sandbox support template if you do not have one ready.

  2. Add Zoekt by using the Zoekt helm chart Setup the GitLab Zoekt Helm chart as a standalone chart or subchart of the GitLab Helm chart

    1. Enable the Zoekt chart
    2. Set CPU and memory usage
    3. Configure Zoekt in GitLab

  3. Test the configuration by running the rake task in the toolbox

    kubectl exec <toolbox pod name> -it -c toolbox -- gitlab-rake gitlab:zoekt:info
    • Confirm Exact Code Search section shows indexing and searching as enabled
    • Under Nodes - that all nodes show as Online
    • Under Indexing status - Confirm that Repositories count has no failed repositories
    • Under Node Details confirm that the version of both nodes is the same and that both are Online.

  4. Use the troubleshooting documentation if there are any failures, and check sidekiq or zoekt logs for further troubleshooting.

Option 1B: Using Helm chart with GitLab Omnibus

This option assumes you already have an Omnibus setup and want to connect Zoekt to GitLab.

Prerequisites: Create a cluster, or use an existing - make sure it's in the same VPC and subnet as your GitLab instance, otherwise choose to be a neworking guru to make it work.

  1. Follow the documentation to connect to your GitLab Omnibus instance.
  2. Confirm that code search works on both group and project levels. To search, go to the "Search or go to" bar at the top left corner. Code search terms should indicate Exact code search (powered by Zoekt) is enabled at the top if all is setup correctly. working_zoekt_search.png

Option 2: Docker Setup (This method is not for production use)

  1. First install Zoekt to have at least one Zoekt node connected to the GitLab instance before enabling exact code search in the Admin settings.

    1. Install docker compose

    2. Install docker and verify it works with sudo docker run hello-world

    3. Clone the docker-compose repo

    4. [Optional] Move the docker compose example to the dir of your choice

      mv example/docker-compose ~/zoekt-docker-compose

    5. Copy the .env file as per the instructions.

      cd ~/zoekt-docker-compose
cp example.env .env

  2. In the .env file, update the GitLab URL to your GitLab instance URL and the GITLAB_ZOEKT_VERSION to match your GitLab server version:

    GITLAB_URL="http://mytest-gitlab.com"
GITLAB_ZOEKT_VERSION=v17.11.3
    • Update docker-compose.yml to set the path for gitlab_shell_secret 
       volumes:
   - ./zoekt_index_data:/data/index
   - /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret:/.gitlab_shell_secret

    Start Zoekt nodes with docker-compose up and specify the UID of gitlay.socket by running ls -ln in the path /var/opt/gitlab/gitaly

    UID=1000 GID=1000 docker-compose up -d

  3. Confirm search is now working with Zoekt. To search, go to the "Search or go to" bar at the top left corner.

    Code search terms should indicate Exact code search (powered by Zoekt) is enabled at the top if all is setup correctly. working_zoekt_search.png Click on "View syntax options" to review the available syntax examples:

    zoekt_syntax_options.png

    Use the troubleshooting documentation if there are any failures, and check sidekiq or zoekt logs for further troubleshooting.

Stage 3: Tickets

  1. Find 10 Solved tickets to get a sense of what gets asked and how others have done troubleshooting in this area.
    1. __
    2. __
    3. __
    4. __
    5. __
    6. __
    7. __
    8. __
    9. __
    10. __
  2. Answer 5 tickets on this module's topic and paste the links here. Do this even if a ticket seems too advanced for you to answer. Find the answers from an expert and relay them to the customers.
    1. __
    2. __
    3. __
    4. __
    5. __

Penultimate stage: Review

Any updates or improvements needed? If there are any dead links, out of date or inaccurate content, missing content whether in this module or in other documentation, list it below as tasks for yourself! Once ready, have a maintainer or manager review.

  1. Update ...

Final stage: Completion

  1. Have your trainer review your tickets and assessment. If you do not have a trainer, ask an expert to review.
  2. Manager: schedule a call (or integrate into 1:1) to review how the module went.
  3. Submit a MR to update modules and knowledge_areas in your Support Team yaml file with this training module's topic. You will now be listed as an expert in this topic on Skills by Person page.
Edited by Rocky Mongare