Add /admin/search/zoekt APIs for controlling Zoekt rollout

What does this MR do and why?

As part of our rollout of Zoekt we're going to be enabling it 1 namespace at a time. This means that we'll need tooling to simplify this process. Originally I was planning on doing this via the Rails console but I've since learnt any rails console commands require a C2 change request. This requires lots of approvals and someone with access to perform the steps. This seemed quite wasteful so I've subsequently realised this would be better to roll out via chatops. Chatops requires an API to execute the steps so this adds some admin APIs we'd need to run from chatops.

Also these same APIs might end up being useful for other administrators managing Zoekt for their instances or possibly to support a future UI.

NOTE on docs: Since this feature is in early development (alpha) and subject to change until we have done some more internal testing it is minimally documented at https://docs.gitlab.com/ee/user/search/exact_code_search.html . We don't yet have sufficient docs for self-managed on how they would enable this feature so I felt that adding a docs page for this API would probably just cause more confusion so I haven't specifically added anything for this.

See Rollout `search_code_with_zoekt` and `index_cod... (#388519)

New Database queries

1. Find a Zoekt::Shard
2. Find all Zoekt::Shard
3. Find a Zoekt::IndexedNamespace
4. Find all Zoekt::IndexedNamespace for a Zoekt::Shard limit 20

How to set up and validate locally

1. Follow the steps at https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/zoekt.md to enable Zoekt in your local environment
2. Run bin/rspec ee/spec/requests/api/admin/search/zoekt_spec.rb

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #404534 (closed)

added devopsdata stores featureenhancement groupglobal search sectioncore platform typefeature labels

assigned to @DylanGriffith

added backend label

	1 Warning
	featureaddition and featureenhancement merge requests normally have a documentation change. Consider adding a documentation update or confirming the documentation plan with the Technical Writer counterpart. For more information, see: The Handbook page on merge request types. The definition of done documentation.

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	Eugenia Grieff (@egrieff) (UTC+2, 8 hours behind @DylanGriffith)	Sean Arnold (@seanarnold) (UTC+12, 2 hours ahead of @DylanGriffith)
database	Leonardo da Rosa (@l.rosa) (UTC-3, 13 hours behind @DylanGriffith)	Adam Hegyi (@ahegyi) (UTC+2, 8 hours behind @DylanGriffith)

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

added 1107 commits

• af670db7...3190bd4a - 1106 commits from branch master
• 34b32592 - Add /admin/search/zoekt APIs for controlling Zoekt rollout

Compare with previous version

added 1 commit

• 1f1caf54 - Add /admin/search/zoekt APIs for controlling Zoekt rollout

Compare with previous version

marked this merge request as ready

marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

requested review from @john-mason

removed review request for @john-mason

@sdungarwal could you please take the initial backend review here. Looks like John Mason is away a few more days,.

requested review from @sdungarwal

added 1 commit

• 16329f40 - Add /admin/search/zoekt APIs for controlling Zoekt rollout

Compare with previous version

added 1 commit

• c500395b - Add /admin/search/zoekt APIs for controlling Zoekt rollout

Compare with previous version

added 1 commit

• 1ce18036 - Add /admin/search/zoekt APIs for controlling Zoekt rollout

Compare with previous version

approved this merge request

@sdungarwal, thanks for approving this merge request.

This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.

For more info, please refer to the following links:

• rspec predictive jobs
• jest predictive jobs
• merging a merge request.

added pipeline:mr-approved label

added 1 commit

• c1771742 - Add /admin/search/zoekt APIs for controlling Zoekt rollout

Compare with previous version

changed milestone to %15.11

mentioned in commit 45f880fe

added 1 commit

• 45f880fe - Add /admin/search/zoekt APIs for controlling Zoekt rollout

Compare with previous version

changed the description

requested review from @splattael

mentioned in commit gitlab-com/chatops@d281ecc7

mentioned in commit gitlab-com/chatops@b194780c

mentioned in merge request gitlab-com/chatops!368 (merged)

mentioned in commit gitlab-com/chatops@13187890

removed review request for @splattael

added feature flag label

changed the description

mentioned in commit 9fa47553

added 512 commits

• 45f880fe...aacffcfa - 510 commits from branch master
• 9fa47553 - Add /admin/search/zoekt APIs for controlling Zoekt rollout
• b2492ffa - Extend API admin test suite with 401 response code

Compare with previous version

changed the description

added database label

@krasio would you mind giving this a database review. The new queries are all pretty uninteresting basic CRUD operations on tables that only have 1 row in production. I added the query plans to the description at !116650 (merged) . Probably the only interesting one is https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/17769/commands/59265 . I realise I'd need another composite index on shard_id,id to make this as optimally fast as it could be but considering that we are likely only going to have a single row in zoekt_shards for the medium term and a few at most in the long term it just didn't seem worth it. Also this is an admin API that will be used only a few times a week.

requested review from @krasio

@splattael back to you. I've incorporated your suggestions

requested review from @splattael

approved this merge request

added databaseapproved label

removed review request for @krasio

added 1 commit

• 115f4316 - Extend API admin test suite with 401 response code

Compare with previous version

mentioned in commit 1b8da271

added 242 commits

• 115f4316...15c8b617 - 241 commits from branch master
• 1b8da271 - Add /admin/search/zoekt APIs for controlling Zoekt rollout

Compare with previous version

mentioned in commit gitlab-com/chatops@8b653f55

mentioned in commit gitlab-com/chatops@c670bd16

mentioned in commit 373f5a36

mentioned in merge request !117634 (merged)

mentioned in commit e1670e16

approved this merge request

mentioned in issue #407328 (closed)

mentioned in issue #407329

mentioned in issue #407331 (closed)

enabled an automatic merge when the pipeline for 75c282c5 succeeds

resolved all threads

added Enterprise Edition label

mentioned in commit 1cdf07be

mentioned in commit 4692206b

It seems that https://gitlab.com/gitlab-org/gitlab/-/jobs/4113298412 got sucked. Retrying

Edit: See #406626

merged

Hello @DylanGriffith

The database team is looking for ways to improve the database review process and we would love your help!

If you'd be open to someone on the database team reaching out to you for a chat, or if you'd like to leave some feedback asynchronously, just post a reply to this comment mentioning:

@gitlab-org/database-team

And someone will be by shortly!

Thanks for your help!

This message was generated automatically. You're welcome to improve it.

mentioned in commit a4bf629d

added workflowstaging-canary label

added workflowcanary label and removed workflowstaging-canary label

added workflowstaging label and removed workflowcanary label

added workflowproduction label and removed workflowstaging label

mentioned in commit c1640234

added workflowpost-deploy-db-production label and removed workflowproduction label

mentioned in commit 5670f066

mentioned in commit gitlab-com/chatops@ef413497

mentioned in commit gitlab-com/chatops@2e428cf1

mentioned in commit f95a5c98

mentioned in commit gitlab-com/chatops@fa6aa098

added releasedcandidate label

added releasedpublished label and removed releasedcandidate label

mentioned in merge request kubitus-project/kubitus-installer!2061 (merged)

mentioned in issue #389765 (closed)