Draft: WIP: Support large YAML definitions in AI Catalog via object storage

What does this MR do and why?

Adds support for storing large YAML definitions in object storage for the AI Catalog. This allows YAML files >64KB to be stored without hitting JSONB size limits, while keeping parsed metadata in the database for querying.

Key changes:

  • Add ai_catalog_yaml_object_storage feature flag (user-scoped)
  • Add ai_catalog object storage configuration
  • Add YamlDefinitionUploader for storing YAML files
  • Add yaml_definition_file and yaml_definition_store columns to ai_catalog_item_versions
  • Make yaml_definition optional in flow_v2.json schema
  • Update YamlDefinitionParser to conditionally exclude inline yaml_definition when feature flag is enabled
  • Update create/update services to upload YAML when flag enabled
  • Add transparent yaml_definition accessor on ItemVersion that reads from object storage when present, falls back to JSONB Issue: #591638

Walkthrough Video

https://youtu.be/zaZxFsWuh1c

MR Breakdown Plan

This MR is being split into smaller, independently shippable chunks. Each MR can be merged behind the feature flag without breaking existing functionality.

MR Description Dependencies
1a Add yaml_definition_file columns to ai_catalog_... (!228120 - merged) None
1b Add ai_catalog object storage configuration and... (!228122 - merged) None
2 Add YamlDefinitionUploader for AI Catalog YAML ... (!228839) 1b
3 Model layer: Mount uploader, add transparent accessor 1a, 2
4 Service layer & schema: Upload in services, make yaml_definition optional 3

How to set up and validate locally

  1. Enable the feature flag:
    Feature.enable(:ai_catalog_yaml_object_storage, User.find_by(username: 'your_username'))
  2. Create/update a flow - the YAML definition should be stored in object storage instead of inline in the JSONB
  3. Query the flow - yaml_definition should transparently return content from object storage

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist (https://docs.gitlab.com/development/code_review/#acceptance-checklist). It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sam Beckham

Merge request reports

Loading