Add CI config to run scenarios
Adds a CI job that can run the scenarios in pipelines in this project, either in MRs or as a new pipeline run against a branch.
This is part of gitlab-org/gitlab#429619 (closed). The aim of that issue is to allow the scenarios to be run from MRs in gitlab-org/gitlab. This is the first part - see the MR breakdown below for the other parts.
The existing script fetches embeddings from the Vertex AI API and assumes you're logged in via gcloud and have access to the AI Enablement dev project. I don't have access to that project so I used a QA one, gitlab-qa-ai-tests-2924ef. The new CI job knows to use that project via the variable GCP_PROJECT, and it authenticates via a service account using the key file I added to CI variables as a file var GOOGLE_APPLICATION_CREDENTIALS.
How to test
CI
- Start a new pipeline in this MR or via https://gitlab.com/gitlab-com/create-stage/code-creation/code-suggestion-scenarios/-/pipelines/new?ref=ml-response-eval-pipeline
- Check the artifacts for
scores.csv
Locally - new functionality
If you have Docker Compose on your machine you can test these changes locally by running most of what the run_scenarios job runs. This assumes GOOGLE_APPLICATION_CREDENTIALS and GCP_PROJECT are unset and you are logged in via gcloud and have access to the AI enablement dev project.
If you'd like to test the authentication path used in the new CI job, then when you run the manual job you can set GCP_PROJECT and GOOGLE_APPLICATION_CREDENTIALS to appropriate values for your own project (or leave GCP_PROJECT unset and set GOOGLE_APPLICATION_CREDENTIALS to a key file for a service account in the AI enablement project).
You'll also need GITLAB_ACTIVATION_CODE set to an SM cloud license that includes a Code Suggestions add-on purchase. I've added a QA license in CI variables that you can also use (or provide your own).
# The compose file mounts a couple of paths so we need to prepare them or the GDK container won't start.
export CI_PROJECT_DIR=/tmp/cs_test
mkdir -p $CI_PROJECT_DIR/log/gdk $CI_PROJECT_DIR/log/gitlab
chown -R 1000:1000 $CI_PROJECT_DIR/log
# Map gdk.test to your docker host IP address. E.g., if Docker is running locally on your Mac:
echo "$(ifconfig -l | xargs -n1 ipconfig getifaddr) gdk.test" >> /etc/hosts
# Launch and prepare GDK
export GDK_IMAGE=registry.gitlab.com/gitlab-org/gitlab/gitlab-qa-gdk:master
docker compose -f ci/support/e2e/compose.yml build --build-arg GDK_IMAGE=$GDK_IMAGE web
docker compose -f ci/support/e2e/compose.yml up -d web
until docker inspect --format "{{json .State.Health.Status }}" gdk | grep -m 1 "healthy"; do sleep 5 ; done
export GITLAB_ACTIVATION_CODE=<insert code>
docker compose -f ci/support/e2e/compose.yml exec --env GITLAB_ACTIVATION_CODE web gdk rails runner /home/gdk/gitlab-development-kit/scripts/prepare-gitlab.rb
# Run the script
bundle install
EXPORT_TO=scores.csv RAILS_TOKEN=ypCa3Dzb23o5nvsixwPA RAILS_URL=http://gdk.test:3000/api/v4/code_suggestions/completions bundle exec ruby run.rbLocally - regression test
To confirm that run.rb can still be run locally as it was before these changes, be sure that GOOGLE_APPLICATION_CREDENTIALS
is unset and you are authenticated via gcloud auth, and then follow the steps as per the readme:
- set up code suggestions
- clone this repository
- run
EXPORT_TO=scores.csv RAILS_TOKEN=<your local access token> RAILS_URL=http://<your local gdk host>:3000/api/v4/code_suggestions/completions ruby run.rb
Example results
MR breakdown
| Description | MR |
|---|---|
Add CI config to run the evaluation script in code-suggestion-scenarios
|
|
Add a job in gitlab-org/gitlab that runs in MRs and triggers the job added in !19 (merged)
|
gitlab-org/gitlab!137611 (merged) |
| Post the results as a discussion in the MR that triggered the evaluation pipeline | follow-up |
| Link to the results CSV artifact from the MR widget | gitlab-org/gitlab!137611 (merged) |
| Document CI variables, GCP requirements, and maintainance runbook | follow-up |