Skip to content

E2E test: generalize code suggestions webIDE test

Jay McCure requested to merge jmc-web-ide-e2e-test into master

What does this MR do and why?

Resolves #437565 (closed)

This test was fast-quarantined as the suggestion changed from what we expected. LLM responses can change for the same prompt, and we can also change models, the suggestion is non-deterministic. This MR changes the tests to ensure that any suggestion is retrieved. We do this by ensuring that additional text is added to the editor.

In the case of code generation we expect additional lines. In the case of code completion we don't expect additional lines.

How to set up and validate locally

This test currently only runs on staging and prod

GITLAB_QA_USER_AGENT=xxx GITLAB_USERNAME=gitlab-qa GITLAB_PASSWORD=xxx GITLAB_QA_ACCESS_TOKEN=glpat-xxx QA_GITLAB_URL=https://staging.gitlab.com bundle exec rspec qa/specs/features/ee/browser_ui/3_create/web_ide/code_suggestions_in_web_ide_spec.rb
  • remove fast quarantine for spec
Edited by Jay McCure

Merge request reports