Collaboration issue: Migrating Duo Chat tools

Overview

• Blueprint section: https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/prompts_migration/#duo-chat-tools
• Epic: Prompt Migration - Duo Chat Tools (gitlab-org&14431 - closed)

Proposal

This MRs migrate Duo Chat Explain Code tool:

• Execute Duo Chat explain code tool via agents (gitlab-org/gitlab!160252 - merged)
• feat(agents): add prompts for explain code tool (!1132 - merged)

Migrating other tools

The approach taken in the example MR can be implemented for other tools:

Rails side

• Define prompt_migration_#{unit_primitive} feature flag
• Include UseAiGatewayAgentPrompt to the executor of the tool
  • If def unit_primitive method is not defined in the executor it will be nil when the feature flag is disabled and if it's enabled
  • If it's defined, it will always contain the defined value
  • It's done to cover this logic
    • If use_ai_gateway_agent_prompt is enabled - agent is called
    • If it's disabled, unit primitive is checked: if unit primitive is present #{BASE_ENDPOINT}/#{unit_primitive} is called; otherwise, ENDPOINT is called
• Add it_behaves_like 'uses ai gateway agent prompt' to the executor's spec

AI Gateway side

• Define prompts in ai_gateway/prompts/definitions/chat/{tool name} folder
• Example: feat(agents): add prompts for explain code tool (!1132 - merged)

Feature flags

All feature flags should have the prefix prompt_migration_ and should be cleaned up afterward.

• prompt_migration_refactor_code
• prompt_migration_write_tests
• prompt_migration_explain_vulnerability

Evaluation

The MR feat(ci): add manual jobs for evaluations (gitlab-org/ai-powered/eli5!90 - merged) contains CI job prompts:single which allows running evaluation for any migrated prompt. However, we don't have datasets for all tools. We can create them ourselves on Langsmith, right? 👍