fix: vertex ai search datastore is not working

What does this merge request do and why?

This MR fixes the issue that the newly created search app & data store for GitLab documentation semantic search is not working. After the investigation, it turned out that the solution_types=[SolutionType.SOLUTION_TYPE_SEARCH] must be set to the DataStore.

This MR also improves the data store quality by explicitly specifying the schema. This is documented in https://cloud.google.com/generative-ai-app-builder/docs/provide-schema as "Provide the schema as a JSON object". More specifically, setting keyPropertyMapping gains the search performance as it's described in the doc. You can see the evaluation result in !1990 (comment 2367572640).

We'll create v17.9 and 17.10 search app & data store after this fix has been merged via the regular ingestion process.

Related to https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/issues/797+

How to set up and validate locally

1. Check out this feature branch.

2. Run the ingestion locally. Example:

   export GCP_PROJECT_NAME="ai-enablement-dev-69497ba7"
   export SEARCH_APP_NAME="sm-gitlab-docs-test"
   export GITLAB_DOCS_REPO="https://gitlab.com/gitlab-org/gitlab.git"
   export GITLAB_DOCS_REPO_REF="master"
   export GITLAB_DOCS_CLONE_DIR="/tmp/gitlab-org/gitlab"
   export GITLAB_DOCS_JSONL_EXPORT_PATH="/tmp/gitlab-org/gitlab/docs.jsonl"
   export GITLAB_DOCS_WEB_ROOT_URL="https://gitlab.com/help"
   
   make ingest > ingest.log 2>&1

3. Test the newly created search app in GCP console.

Merge request checklist

• Tests added for new functionality. If not, please raise an issue to follow up.
• Documentation added/updated, if needed.

added groupai framework label

assigned to @shinya.maeda

added devopsai-powered sectiondata-science labels

Reviewer roulette

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines.

Once you've decided who will review this merge request, mention them as you normally would! Danger does not automatically notify them for you.

Reviewer	Maintainer
@ck3g (UTC+1, 8 hours behind author)	@achueshev (UTC+1, 8 hours behind author)

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

Proper labels assigned to this merge request. Please ignore me.

@shinya.maeda - please see the following guidance and update this merge request.

	1 Error
	Please add typebug typefeature, or typemaintenance label to this merge request.

added 1 commit

• 1d97affb - fix: vertex ai search datastore is not working

Compare with previous version

Evaluation pipeline for sm-test-gitlab-docs-v6-17-9: https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/evaluation-runner/-/pipelines/1676026239

added typebug label

resolved all threads

added 1 commit

• 00dcd3e8 - fix: vertex ai search datastore is not working

Compare with previous version

Evaluation pipeline for sm-test-gitlab-docs-v9-17-9: https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/evaluation-runner/-/pipelines/1676108541

added 1 commit

• d3a86ba1 - fix: vertex ai search datastore is not working

Compare with previous version

Evaluation pipeline for sm-test-gitlab-docs-v10-17-9: https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/evaluation-runner/-/pipelines/1676135096

changed the description

added 1 commit

• 573e3cdb - fix: vertex ai search datastore is not working

Compare with previous version

added 1 commit

• 905f3c2f - fix: vertex ai search datastore is not working

Compare with previous version

added 1 commit

• 16e4266f - fix: vertex ai search datastore is not working

Compare with previous version

added 59 commits

• 16e4266f...470c46ca - 58 commits from branch main
• df347505 - fix: vertex ai search datastore is not working

Compare with previous version

changed milestone to %17.10

added 9 commits

• df347505...1aa30bdc - 8 commits from branch main
• 59de9c54 - fix: vertex ai search datastore is not working

Compare with previous version

added 1 commit

• 6822ec30 - fix: vertex ai search datastore is not working

Compare with previous version

Evaluation pipeline for !1990 (6822ec30) (with explicit schema): https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/evaluation-runner/-/pipelines/1689066334

added 1 commit

• 62ad936a - fix: vertex ai search datastore is not working

Compare with previous version

added 1 commit

• 67176a2f - fix: vertex ai search datastore is not working

Compare with previous version

Evaluation pipeline for !1990 (67176a2f) (without explicit schema, just setting solution_types): https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/evaluation-runner/-/pipelines/1689079038

changed the description

resolved all threads

resolved all threads

added 1 commit

• a3f64647 - fix: vertex ai search datastore is not working

Compare with previous version

changed the description

@lulalala Would you mind reviewing this fix? Please pass it to @brytannia if it looks good to you. Thanks!

requested review from @lulalala

LGTM! Assigning @brytannia

left review comments without approving or requesting changes

requested review from @brytannia

removed review request for @lulalala

approved this merge request

resolved all threads

enabled automatic add to merge train when checks pass

started a merge train

merged

mentioned in commit 2304d9b3