Skip to content

E2E tests: add SM code suggestion scenarios

Jay McCure requested to merge jmc-cs-sm-tests into master

What does this MR do and why?

Adds additional self-managed code suggestion E2E api tests. These tests will run in MRs.

  • code generation
  • code generation with streaming

Resolves https://gitlab.com/gitlab-org/quality/quality-engineering/team-tasks/-/issues/2301

Other fixes:

  • fixes completion test to use a longer code prompt, as the previous prompt was getting code generation, so currently we are not invoking completion
  • log the output of the response (useful for debugging incidents)
  • language_identifier has been moved in the request to match where it actually is
  • better assertion to show the expected status code
  • some refactoring

It turns out that our completion test was actually using generation due to the SMALL_FILE_TRIGGER, i.e our prompt was too small so it would revert to generation.

To test the self-managed version of the test locally:

CHROME_DISABLE_DEV_SHM=true QA_IMAGE="registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:jmc-cs-sm-tests"  DOCKER_DEFAULT_PLATFORM=linux/amd64    GITLAB_LICENSE_MODE=test QA_EE_ACTIVATION_CODE= GITLAB_QA_ACCESS_TOKEN= bundle exec gitlab-qa Test::Integration::AiGateway EE

Reference: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#testintegrationaigateway-eefull-image-address

To verify on staging: GITLAB_QA_USER_AGENT=x GITLAB_USERNAME=gitlab-qa GITLAB_PASSWORD=x GITLAB_QA_ACCESS_TOKEN=x QA_GITLAB_URL=https://staging.gitlab.com bundle exec rspec qa/specs/features/ee/api/3_create/code_suggestions_spec.rb --tag external_ai_provider

Edited by Jay McCure

Merge request reports