Create a sustainable 'feature flag' option for AI Gateway prompt changes

Problem to solve

The current method for making changes to prompts, for example the Duo Chat ReAct prompt is to create a new .yml file, copying everything over from the previous prompt. Then contributors should make a feature flag (FF) in the Rails repository and specify the new file as being used.

Thanks to @alejandro for the explanation:

  • Create a new yml file in the AIGW with the changes (say we name it base-v2.yml )
  • Switch between the current and the new yml using FFs:
#ee/lib/gitlab/llm/ai_gateway/completions/rewrite_description.rb
module Gitlab
  module Llm
    module AiGateway
      module Completions
        class RewriteDescription < Base
          def agent_name
            Feature.enabled?(:my_ff) ? 'base-v2' : 'base'
          end

The problems I've encountered with this are:

  1. The prompt change process is already fairly long and difficult. Now people also have to coordinate code changes in 2 separate repos.

  2. Sometimes the base prompt file changes, as happened here. I created my MR, and by the time it was ready for maintainer review, the original prompt file had already changed. It will be very hard to keep them in sync.

  3. People have suggested prompt versions as a way to fix this, but even with prompt versions, there has to be a way to switch between them in a way similar to feature flags.

Note: I noticed this because of the Duo Chat ReAct prompt, but it would be a problem for any other prompts in the AI Gateway as well.

Proposal

I propose that we use a feature flag system that is fully maintained within the AI Gateway. https://github.com/Pytlicek/fastapi-featureflags

Definitely open to other suggestions though!

We should use GitLab Feature Flag https://docs.gitlab.com/ee/operations/feature_flags.html with Unleash python client.

Definition of done

Make quick research of possible solutions and implement chosen one. Document it in Duo Chat dev guide

Edited by Shinya Maeda