[GitLab-Rails] Add feature test for the entire flow (use AI GW in CI)
Problem to solve
On the GitLab-Rails side, we currently don't have a test that covers that entire flow in a black-box approach.
We are adding something similar for AI GW currently - #503 (closed) - but we want to expand it and add a similar test to GitLab-Rails monolith.
The test should use AI GW in Rails CI, ask for a use token, and then use the obtained token to successfully request Code Suggestions /completions
.
This will allow us to catch cross-level and cross-service integration issues that are hard to notice with lower-level tests.
Proposal
We recently started using AI Gateway in the CI jobs in GitLab-Rails. This allows us to continuously verify that client requests (e.g. from Sidekiq jobs) are correctly formatted and AI Gateway can correctly respond to it. In fact, we discovered a couple of bugs in the past, like https://gitlab.slack.com/archives/C051K31F30R/p1717479004054009. Here are the Ci jobs that currently requesting to AI Gateway.
I wonder if we can set AIGW_AUTH__BYPASS_EXTERNAL
to false
here, so that we can test OIDC as well. You can also add a new test case for code suggestions + user-level JWT. To do so, we would need to tweak the job/rspec settings to let AI Gateway service access to Discovery Endpoint (/.well-known/openid-configuration
) and JWKS URI. Though, I'm not sure how difficult it is, because in the job most likely Rails server is not running. Or, maybe we can use Capybara..?
Further details
TBD