Create pipeline/s to run AI E2E tests which use a real AI provider
From: https://gitlab.com/gitlab-com/gitlab-OKRs/-/work_items/5920#note_1852163767
general
-
Verify environment is up and running -
Ensure smoke tests (non-AI) pass https://gitlab.com/gitlab-org/quality/gitlab-qa-executor/-/pipelines/1248588495 - [ ] Identify/create user which has access to AI servicesN/A
gitlab tests
-
Add tags to gitlab tests identifying e2e tests that invoke the external ai provider gitlab-org/gitlab!149032 (merged) -
Verify manually that Code Suggestions works in environment under test -
Verify manually that Duo Chat works in environment under test -
Modify/clone https://gitlab.com/gitlab-org/quality/gitlab-qa-executor (additional env variables may need to be added) - [ ] Configure https://gitlab.com/gitlab-org/quality/gitlab-qa-executor with ai userN/A -
Run AI tests -> Passing run ✅ https://gitlab.com/gitlab-org/quality/gitlab-qa-executor/-/jobs/6644577268#L349 -
Document instructions -> #2578 (comment 1875938656)
ide tests (vscode and jetbrains)
-
Verify manually that the extension works when configured to use environment under test -
Add additional pipeline schedule for new environment using AI token from the environment -
Run tests (ad hoc?/schedule?) (if test is to be run many times/long term, we may have to consider running it outside of the project as it will occur costs for the project)
Edited by Jay McCure