Draft: WIP: Support large YAML definitions in AI Catalog via object storage
What does this MR do and why?
Adds support for storing large YAML definitions in object storage for the AI Catalog. This allows YAML files >64KB to be stored without hitting JSONB size limits, while keeping parsed metadata in the database for querying.
Key changes:
- Add
ai_catalog_yaml_object_storagefeature flag (user-scoped) - Add ai_catalog object storage configuration
- Add YamlDefinitionUploader for storing YAML files
- Add
yaml_definition_fileandyaml_definition_storecolumns toai_catalog_item_versions - Make
yaml_definitionoptional inflow_v2.jsonschema - Update YamlDefinitionParser to conditionally exclude inline yaml_definition when feature flag is enabled
- Update create/update services to upload YAML when flag enabled
- Add transparent
yaml_definitionaccessor on ItemVersion that reads from object storage when present, falls back to JSONB Issue: #591638
Walkthrough Video
MR Breakdown Plan
This MR is being split into smaller, independently shippable chunks. Each MR can be merged behind the feature flag without breaking existing functionality.
| MR | Description | Dependencies |
|---|---|---|
| 1a | Add yaml_definition_file columns to ai_catalog_... (!228120 - merged) | None |
| 1b | Add ai_catalog object storage configuration and... (!228122 - merged) | None |
| 2 | Add YamlDefinitionUploader for AI Catalog YAML ... (!228839) | 1b |
| 3 | Model layer: Mount uploader, add transparent accessor | 1a, 2 |
| 4 | Service layer & schema: Upload in services, make yaml_definition optional | 3 |
How to set up and validate locally
- Enable the feature flag:
Feature.enable(:ai_catalog_yaml_object_storage, User.find_by(username: 'your_username')) - Create/update a flow - the YAML definition should be stored in object storage instead of inline in the JSONB
- Query the flow - yaml_definition should transparently return content from object storage
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist (https://docs.gitlab.com/development/code_review/#acceptance-checklist). It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Sam Beckham