AI Catalog support for interoperability with hard coded Workflow Service flows

About

AI Catalog is allowing Flows and Agents to be defined in the GitLab Rails Monolith.

Problem

The limitations of the flows and agents defined in the AI Catalog hinge on our ability to support the rich flow definitions possible within Workflow Service (see Flow Registry docs) within AI Catalog backend and frontend.

Likely to be the case in beta, or even early GA, our catalog's support for complex flow definition will be "behind" the capabilities of a hard-coded flow. As an example conditional ordering of agent execution.

We want to support these richer hard-coded flows while AI Catalog capabilities are improving. And we want the AI Catalog to remain the mechanism for how customers enable flows for projects, even those that are hard-coded.

Proposal 1: Source of Truth is AI Catalog

Allow AI Catalog flows to be optionally defined in pure raw Flow Registry data, as per the Flow Registry doc.

This could be behind a feature flag, only enabled for gitlab-org group on GitLab.com, so on GitLab.com we limit the ability to just GitLab team members to create these flows.

This would allow anything that can be defined in Flow Registry to be defined in AI Catalog.

• New column to save this alternative flow definition.
• Could be behind a feature flag, only enabled for gitlab-org group, to limit the ability to just GitLab team members to create on GitLab.com
• Raw YAML/JSON pasted into form (visible when flag enabled) and saved by mutation (allowed when flag enabled) as a new version of the flow in place of the regular configuration.
• This data would be sent in place of the flow configuration we currently generate when it is present in a flow.

Proposal 2: Source of Truth is Flow Registry

We would support an AI Catalog flow to be a hard-coded type. When triggered, a specific hard-coded Workflow Service flow would run.

This could be behind a feature flag, only enabled for gitlab-org group on GitLab.com, so on GitLab.com we limit the ability to just GitLab team members to create these flows.

• Backend-only support for a special kind of AI Catalog flow, where it does not have any configuration data itself, but names a particular hard-coded Workflow Service flow.
• Could be behind a feature flag, only enabled for gitlab-org group, to limit the ability to just GitLab team members to create on GitLab.com
• A customer could enable this type of flow for their project from the Catalog as normal.
• But no version pinning options would be available for these flows - as they could only run at the "latest" version
• When triggered, the backend would treat it as a flow defined in Workflow Service and not build the flow data itself, but instead, just ask Workflow Service to run the hard-coded flow. Note, this functionality exists in Rails already, our AI Catalog does not yet support that interoperability though.