Add the Project Telescope design document

---

title: "Telescope: the engine underneath"

description: "Why Telescope builds on Zoekt: how the engine works, which limits are configuration and which are engine design, and what would have to change later."

toc_hide: true

---

<!-- Design Documents often contain forward-looking statements -->

<!-- vale gitlab.FutureTense = NO -->

This page is an appendix to the [technical design](/handbook/engineering/architecture/design-documents/telescope/technical-design/) for [Project Telescope](/handbook/engineering/architecture/design-documents/telescope/). It answers one question: why build on Zoekt, a ten-year-old open source engine, rather than build or buy something newer. Bracketed numbers point to the technical design's [reference list](/handbook/engineering/architecture/design-documents/telescope/technical-design/#references).

## How Zoekt works

Zoekt is a trigram index. For every file it records each three-character sequence and a posting list of the files and positions where that sequence occurs. A search for `PaymentProcessor` first intersects the posting lists for the term's fourteen overlapping trigrams (`Pay`, `aym`, `yme`, and so on). That reduces millions of files to a handful of candidates, and only those candidates get the real match.

The same design explains both the speed and the limits:

- The median exact query completes inside the webserver in about 100 ms (p50 119 ms on GitLab.com). The slow tail (p95 1,206 ms) comes from how the engine is deployed, with one namespace per node on network disk and indexing on the same node as serving, not from the data structure \[2\].

- A regular expression with no literal text gives the index nothing to narrow with, so the engine scans every document.

- A file with an unusually large number of distinct trigrams, typically minified or generated code, bloats every posting list it appears in. Such files are indexed by filename only.

### What a shard contains

A shard is one file on local disk holding the index for one repository (a very large repository is split into a few). It contains the file contents, the trigram posting lists, and metadata: filenames, branches, and symbols. Zoekt does not read the shard with ordinary file reads. It memory-maps it, which means the operating system maps the file into the process's memory and loads pages from disk only when a query touches them, then keeps the frequently used pages in RAM. A repository with 40,000 files produces a shard of a few hundred megabytes. The first query after a node restarts pays for disk reads; the queries that follow are answered from memory, which is why the difference between local SSD and network disk shows up in the tail latency rather than the median.

Each stored file version carries a 64-bit branch mask: one bit per indexed branch of that repository. If `config/database.yml` is identical on `main`, `release-17-4`, and `feature/payment-retry`, it is stored once with three bits set. When the feature branch changes the file, a second version is stored with only the feature branch's bit set. A search scoped to `release-17-4` considers only versions whose bit for that branch is on. This is why indexing more branches costs far less than the number of branches multiplied by the size of the repository: most files are identical on most branches. The technical design uses this under [Which branches](/handbook/engineering/architecture/design-documents/telescope/technical-design/#which-branches).

At indexing time Zoekt runs ctags over each file and stores the names of the functions, classes, and methods it defines. That is what the `sym:` operator searches. `sym:retry` matches the file where `retry` is defined as a method and skips a comment that says "we retry three times" or a log line that mentions retrying; a plain text search for `retry` returns all three. Matches on symbol definitions also score higher in ranking. ctags reads the code's structure without compiling it, so it can tell a definition from a mention but cannot tell two `retry` methods in different classes apart. That gap is why the technical design adds a precise symbol index built from SCIP; see [Symbols](/handbook/engineering/architecture/design-documents/telescope/technical-design/#symbols).

### Where Zoekt comes from

Zoekt was written at Google in 2016, modeled on Google's internal code search. Sourcegraph has maintained the main fork since 2017 \[14\], and GitLab runs a current build of its own fork \[14\].

## What other code hosts run

- Google runs its internal code search over its monorepo. Zoekt is the open-source descendant of that work.

- GitHub tried Solr and then Elasticsearch, and between 2020 and 2023 built its own engine in Rust. It uses content-addressed sharding, so a file that appears in ten thousand forks is indexed once; commit-delta indexing, so only what a commit changed is indexed; and variable-length sparse n-grams, so common sequences do not produce enormous posting lists. In 2023 it indexed 45 million repositories into the index described under [Capacity, sizing, and hardware](/handbook/engineering/architecture/design-documents/telescope/technical-design/#capacity-sizing-and-hardware) \[8\].

- Sourcegraph runs Zoekt beside a SCIP store \[15\].

- IDE agents generally grep the local checkout. That works well for one repository on one machine and does not extend beyond it.

## Configuration problems versus engine limits

Most of the failures described in the proposal's journeys come from GitLab's wrapper around Zoekt or from how it is deployed, not from the engine:

- Indexing only the default branch is our policy, not the engine's limit. It was chosen to hold down index size and infrastructure cost, and this plan does not treat branch indexing as free either; the spike's load measurements exist to price it. Zoekt indexes up to 64 branches per repository. The ceiling is real, though. It is one bit per branch in a 64-bit mask, so no setting raises it, and a repository with a thousand open merge requests cannot hold a slot for each one. The policy fills the slots by priority, reports what it left out, and for repositories of that shape splits one repository into several logical Zoekt repositories by branch group; see [Which branches](/handbook/engineering/architecture/design-documents/telescope/technical-design/#which-branches).

- The size, trigram, and file-count limits are configuration values. Reporting what they skipped costs nothing.

- The Rails wrapper adds at least 141 ms mean per call: per-request authorization computation, connection setup, fan-out, redaction, and rendering \[2\]. The deployment adds the rest of the tail: one namespace per node, indexing beside serving, network disk. Sourcegraph's documentation describes sub-second results on the same engine, served through a small fan-out service that streams results as they arrive \[15\].

- Redaction is a large share of that 141 ms on its own. It runs per result, and its cost follows the caller's membership graph rather than the size of the result set, so a user in a large group pays more than the result count suggests. Pushing the rules into the query is what removes that pass; where a rule cannot become a filter, the plane keeps the redaction step and says so on the response.

- Symbol ranking is enabled in GitLab's build. Whether BM25F is present and enabled is a configuration check.

Three limits are in Zoekt itself, and no setting changes them:

1. Shards are per repository, not per unique file. Deduplicating across repositories needs the content-addressed layout, designed together with location-level authorization.

1. Compaction. Pushes are already indexed incrementally, but the periodic force reindex that folds delta shards back together has to be bounded on large repositories.

1. Shards live only on local disk. Zoekt has no concept of a durable copy elsewhere, so the fleet is sized for the whole index, hot and cold, and a lost node means a rebuild from Gitaly. Keeping finished shards in object storage and pulling them to local disk is placement logic around Zoekt rather than a format change, so the fork risk is low. Reading cold shards from object storage at query time is not proposed; see [lesson 7 of the lessons carried in](/handbook/engineering/architecture/design-documents/telescope/technical-design/#appendix-lessons-carried-in).

One more limit to monitor: trigram selectivity degrades as the corpus grows, and GitHub reached that point at hundreds of terabytes of content \[8\]. GitLab.com's unique content is well below that. The gateway's p95 and the shard-heat measurement will show whether it starts to matter.

## Compressing content sections: considered and rejected

Zoekt keeps file content uncompressed in the shard so that it can verify regular expressions and produce snippets. Block compression was considered and is not in the plan. It competes with the memory-mapped read path, adds CPU to a path that already measures 248 ms mean per shard search, changes the shard format (a second fork surface), and is largely redundant if deduplication lands. We would revisit it only if the measured duplication ratio comes back low.

## Roadmap

This sequence depends on the measurements described in the technical design and is subject to the disclaimer on the [proposal](/handbook/engineering/architecture/design-documents/telescope/).

- **Next two quarters.** Turn on what already exists: branch indexing, BM25F once verified, and reporting of every skip. Take the measurements that decide the later steps. Contribute skip accounting and branch policy upstream through GitLab's open source contribution process, since both are generic.

- **Next two quarters, and larger than it looks: move the serving path out of Rails.** This is a migration with its own transition and test plan, not a refactor. The Search team's [zoekt-orchestrator](https://gitlab.com/gitlab-org/search-team/zoekt-orchestrator) project is where that work is taking shape; it is not an official project yet.

- **Next year.** The object-storage durability layer, if the decision request is approved. The content-addressed layout, if the duplication ratio justifies it, built in the open. The precise SCIP tier.

- **Eighteen months and beyond.** Evaluate sparse grams on a copy of a shard, under license review. Hold an explicit checkpoint on whether a different engine is ever needed. If it is, that is a multi-year project with a dedicated team, which is what it took GitHub. The request contract exists so that swapping the engine would not mean redoing the plane.