Add support for flow registry v1 version-based definitions for DAP custom flows

What does this MR do and why?

This merge request introduces backend support for a YAML-based editor for DAP flows as an interim solution while the full flow editor is under development. This feature enables users to create and edit flows directly using YAML definitions from the flow registry, without relying on catalog agent references. The syntax adheres to the specifications defined in the Flow Registry v1 documentation.

NOTE: Currently, this flow is enabled for only a few internal team members. We have requested these members to delete their flows to avoid migrating them to the new definition-based flow - see internal slack. In GDK, run the script below to remove data related to the old flow.

flows = Ai::Catalog::Item.where(item_type: Ai::Catalog::Item::FLOW_TYPE)

Ai::Catalog::Item.transaction do 
  Ai::Catalog::ItemConsumer.where(ai_catalog_item_id: flows.ids).destroy_all
  Ai::Catalog::ItemVersionDependency.where(dependency_id: flows.ids).destroy_all
  
  flows.update_all(latest_version_id: nil, latest_released_version_id: nil)
  
  Ai::Catalog::ItemVersion.where(ai_catalog_item_id: flows.ids).destroy_all
  
  flows.destroy_all
end

Issue - #578228 (closed)

How to set up and validate locally

1. Enable FF ai_catalog_flows
2. Create Flow via graphql request below

mutation create_flow($yaml_string: String) {
  aiCatalogFlowCreate(
    input: {name: "Issue Planner Flow", description: "A pirate doing a sum", projectId: "gid://gitlab/Project/1000000", public: true,  addToProjectWhenCreated:true, definition: $yaml_string}
  ) {
    errors
  item {
      id
      versions {
        count
        nodes {
          ...FlowVersionFragment
        }
      }
    }
  }
}

fragment FlowVersionFragment on AiCatalogFlowVersion {
  steps {
    edges {
      node {
        pinnedVersionPrefix
      }
    }
  }
  definition  
}

YAML Flow based on Flow Registry Framework v1 (use below as a graphql variable)

{
  "yaml_string":  "version: \"v1\"\nenvironment: ambient\n\ncomponents:\n  # Step 1: Parse the goal to extract issue ID\n  - name: \"parse_goal\"\n    type: AgentComponent\n    prompt_id: \"goal_parser\"\n    inputs:\n      - from: \"context:goal\"\n        as: \"user_goal\"\n    toolset: []\n    ui_log_events:\n      - \"on_agent_final_answer\"\n\n  # Step 2: Read the issue details\n  - name: \"read_issue\"\n    type: DeterministicStepComponent\n    tool_name: \"get_issue\"\n    inputs:\n      - from: \"context:project_id\"\n        as: \"project_id\"\n      - from: \"context:parse_goal.final_answer\"\n        as: \"issue_iid\"\n    ui_log_events:\n      - \"on_tool_execution_success\"\n      - \"on_tool_execution_failed\"\n\n  # Step 3: Analyze the issue and create a comprehensive plan\n  - name: \"analyze_and_plan\"\n    type: AgentComponent\n    prompt_id: \"issue_plan_analyzer\"\n    inputs:\n      - from: \"context:read_issue.tool_responses\"\n        as: \"issue_data\"\n      - from: \"context:project_id\"\n        as: \"project_id\"\n    toolset:\n      - \"get_repository_file\"\n      - \"list_repository_tree\"\n      - \"find_files\"\n      - \"blob_search\"\n      - \"list_commits\"\n      - \"get_commit\"\n      - \"documentation_search\"\n    ui_log_events:\n      - \"on_tool_execution_success\"\n      - \"on_tool_execution_failed\"\n      - \"on_agent_final_answer\"\n\n  # Step 4: Update the issue with the generated plan\n  - name: \"update_issue_with_plan\"\n    type: DeterministicStepComponent\n    tool_name: \"update_issue\"\n    inputs:\n      - from: \"context:project_id\"\n        as: \"project_id\"\n      - from: \"context:parse_goal.final_answer\"\n        as: \"issue_iid\"\n      - from: \"context:analyze_and_plan.final_answer\"\n        as: \"description\"\n    ui_log_events:\n      - \"on_tool_execution_success\"\n      - \"on_tool_execution_failed\"\n\nprompts:\n  - name: \"goal_parser\"\n    prompt_id: \"goal_parser\"\n    model:\n      params:\n        model_class_provider: anthropic\n        model: claude-sonnet-4-20250514\n        max_tokens: 1024\n    prompt_template:\n      system: |\n        You are a goal parser that extracts issue IDs from natural language requests.\n        \n        Your task is to identify and extract the issue ID (issue_iid) from user goals.\n        \n        Examples of valid goals:\n        - \"please update issue description on issue - 123\" → extract: 123\n        - \"update issue 456 description\" → extract: 456\n        - \"modify issue #789\" → extract: 789\n        - \"work on issue 101\" → extract: 101\n        \n        Return ONLY the numeric issue ID, nothing else.\n        If no issue ID is found, return \"ERROR: No issue ID found in goal\"\n      user: |\n        Extract the issue ID from this goal: {{user_goal}}\n      placeholder: history\n    params:\n      timeout: 30\n\n  - name: \"issue_plan_analyzer\"\n    prompt_id: \"issue_plan_analyzer\"\n    model:\n      params:\n        model_class_provider: anthropic\n        model: claude-sonnet-4-20250514\n        max_tokens: 32_768\n    prompt_template:\n      system: |\n        You are GitLab Duo, an AI assistant specialized in project planning and issue analysis.\n        Your role is to analyze GitLab issues and create comprehensive, actionable plans.\n        \n        When analyzing an issue, you should:\n        1. Understand the problem or feature request thoroughly\n        2. Explore the codebase to understand the current implementation\n        3. Research relevant documentation and best practices\n        4. Create a detailed, step-by-step implementation plan\n        5. Include technical considerations, potential challenges, and testing strategies\n        \n        Your response should be a well-structured issue description that includes:\n        - Clear problem statement or feature description\n        - Technical analysis of the current state\n        - Detailed implementation plan with numbered steps\n        - Acceptance criteria\n        - Testing considerations\n        - Potential risks and mitigation strategies\n        \n        Format your response as a complete issue description in Markdown format that will replace the existing description.\n      user: |\n        Please analyze this GitLab issue and create a comprehensive plan for implementation.\n        \n        Issue Data: {{issue_data}}\n        Project ID: {{project_id}}\n        \n        Analyze the codebase, understand the context, and provide a detailed implementation plan.\n      placeholder: history\n    params:\n      timeout: 300\n\nrouters:\n  - from: \"parse_goal\"\n    to: \"read_issue\"\n  - from: \"read_issue\"\n    to: \"analyze_and_plan\"\n  - from: \"analyze_and_plan\"\n    to: \"update_issue_with_plan\"\n  - from: \"update_issue_with_plan\"\n    to: \"end\"\n\nflow:\n  entry_point: \"parse_goal\"",
  "mr_reviewer":  "version: \"v1\"\nenvironment: ambient\ncomponents:\n  # Step 1: Fetch and format all MR data\n  - name: \"build_review_context\"\n    type: DeterministicStepComponent\n    tool_name: \"build_review_merge_request_context\"\n    inputs:\n      - from: \"context:project_id\"\n        as: \"project_id\"\n      - from: \"context:goal\"\n        as: \"merge_request_iid\"\n    ui_log_events:\n      - \"on_tool_execution_success\"\n      - \"on_tool_execution_failed\"\n\n  # Step 2: Pre-scan codebase for additional context\n  - name: \"prescan_codebase\"\n    type: AgentComponent\n    prompt_id: \"code_review_prescan\"\n    prompt_version: \"^1.0.0\"\n    inputs:\n      - from: \"context:project_id\"\n        as: \"project_id\"\n      - from: \"context:goal\"\n        as: \"merge_request_iid\"\n    toolset:\n      - \"build_review_merge_request_context\"\n      - \"list_repository_tree\"\n      - \"get_repository_file\"\n      - \"read_file\"\n      - \"find_files\"\n      - \"blob_search\"\n    ui_log_events:\n      - \"on_tool_execution_success\"\n      - \"on_tool_execution_failed\"\n      - \"on_agent_final_answer\"\n\n  # Step 3: Perform code review\n  - name: \"perform_code_review\"\n    type: AgentComponent\n    prompt_id: \"review_merge_request\"\n    prompt_version: \"^2.0.0\"\n    inputs:\n      - from: \"context:build_review_context.tool_responses\"\n        as: \"mr_data\"\n      - from: \"context:prescan_codebase.final_answer\"\n        as: \"codebase_context\"\n      - from: \"context:current_date\"\n        as: \"current_date\"\n    toolset: []\n\n  # Step 4: Publish code review\n  - name: \"publish_review\"\n    type: DeterministicStepComponent\n    tool_name: \"post_duo_code_review\"\n    inputs:\n      - from: \"context:perform_code_review.final_answer\"\n        as: \"review_output\"\n      - from: \"context:project_id\"\n        as: \"project_id\"\n      - from: \"context:goal\"\n        as: \"merge_request_iid\"\n    ui_log_events:\n      - \"on_tool_execution_success\"\n      - \"on_tool_execution_failed\"\n\nrouters:\n  - from: \"build_review_context\"\n    to: \"prescan_codebase\"\n  - from: \"prescan_codebase\"\n    to: \"perform_code_review\"\n  - from: \"perform_code_review\"\n    to: \"publish_review\"\n  - from: \"publish_review\"\n    to: \"end\"\n\nflow:\n  entry_point: \"build_review_context\""
}

3. Set up flow triggers for this flow and trigger it from an issue comment.
4. Update flow via graphql

mutation updateFlow($yaml_string: String) {
  aiCatalogFlowUpdate(
    input: {id: "gid://gitlab/Ai::Catalog::Item/206", definition: $yaml_string}
  ) {
    errors
    item {
      id
      name
      description
      public
      versions {
        count
        nodes {
          ...FlowVersionFragment
        }
      }
    }
  }
}

5. Verify Classic Agentic chat working on Web and IDE.
6. Verify Agentic chat with custom agents working on Web and IDE.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #578228 (closed)