Skip to content

Implement Semantic Versioning for Flow Registry YAML Config Files

Problem Statement

Flow Registry YAML configuration files currently lack a robust versioning mechanism similar to the Prompt Registry, which creates significant challenges in maintaining backwards compatibility and supporting AI-powered feature development:

Current State Issues

  1. Version Declaration Without Enforcement: Flow configs declare a version attribute (version: "v1" or version: "experimental"), but this only indicates which Flow Registry framework version to use, not the specific version of the flow configuration itself.

  2. No Granular Config Versioning: Individual flow configurations cannot be versioned independently. When a flow config needs updates (e.g., new components, modified routing logic, updated inputs), there's no way to:

    • Maintain multiple versions of the same flow simultaneously
    • Allow gradual migration of consumers from old to new versions
    • Roll back to previous working configurations if issues arise
    • Test new configurations in production while keeping stable versions available

  3. Breaking Changes Impact All Consumers: Any modification to a flow config (e.g., code_review.yml) immediately affects all consumers, making it risky to:

    • Experiment with improvements
    • Add new capabilities that might need testing
    • Refactor component structures
    • Update prompt versions used by components

  4. Difficult Backwards Compatibility: Unlike Prompt Registry, where prompts can specify version ranges (e.g., prompt_version: "^1.0.0"), flow configs have no mechanism to:

    • Specify compatibility requirements
    • Support semantic version resolution
    • Allow consumers to pin to specific versions or version ranges

  5. Unclear Configuration Evolution: Without semantic versioning, it's difficult to:

    • Communicate the nature of changes (breaking vs non-breaking)
    • Understand when it's safe to adopt updates
    • Maintain changelog clarity
    • Track configuration maturity (stable vs experimental)

Comparison with Prompt Registry

The Prompt Registry successfully implements semantic versioning:

# Prompt Registry structure
ai_gateway/prompts/definitions/
  review_merge_request/
    base/
      1.0.0.yml  # Initial stable version
      1.1.0.yml  # New features added
      1.2.0.yml  # More enhancements
      1.3.0.yml  # Additional improvements
      1.4.0.yml  # Latest version
    claude_3/
      1.0.0.yml
    gpt/
      1.0.0.yml

This allows:

  • Multiple versions to coexist
  • Semantic version range resolution (e.g., ^1.0.0 matches any 1.x.x version)
  • Clear communication of changes through version numbers
  • Gradual migration paths for consumers
  • Safe experimentation with new versions

Current Flow Registry Structure

# Current Flow Registry structure
duo_workflow_service/agent_platform/
  v1/
    flows/
      configs/
        code_review.yml        # Only one version available
        prototype.yml
  experimental/
    flows/
      configs/
        code_review.yml        # Different flow, but no version control
        fix_pipeline.yml
        resolve_sast_vulnerability.yml

Proposed Solution

Implement semantic versioning for Flow Registry configurations following the Prompt Registry pattern, with the following structure:

Directory Structure

duo_workflow_service/agent_platform/
  v1/
    flows/
      configs/
        code_review/
          1.0.0.yml  # Initial stable release
          1.1.0.yml  # Backward-compatible enhancement
          1.2.0.yml  # Another enhancement
          2.0.0.yml  # Breaking changes
        fix_pipeline/
          1.0.0.yml
          1.1.0.yml
        resolve_sast_vulnerability/
          1.0.0.yml
  experimental/
    flows/
      configs/
        code_review/
          0.1.0-dev.yml     # Development version
          0.2.0-alpha.yml   # Alpha testing
        new_flow/
          0.1.0-dev.yml     # Experimental flow

Open questions

  1. How to pass flow configs version information with startWrokflowRequest message?
    1. Should the current pattern for workflow_definition field: <relative_path_to_flows_config>/<flow_registry_version> be expanded to <relative_path_to_flows_config>/<flow_version>/<flow_registry_version>
    2. Should a new filed be added into startWorkflowRequst message?
  2. How backwards compatibility will be assured to already existing versionless flows?
  3. Should instead a set of new field for Flow Registry be added (flow_identifier, flow_version, plus flowConfigSchemaVersion should be used for all flows)?

References

  • Prompt Registry Versioning: docs/aigw_prompt_registry.md
  • Flow Registry Documentation: docs/flow_registry/index.md
  • Semantic Versioning Specification: https://semver.org/
  • Prompt Registry Implementation: ai_gateway/prompts/definitions/
  • Current Flow Configs: duo_workflow_service/agent_platform/{v1,experimental}/flows/configs/

Related Issues

This issue relates to the broader Duo Agent Platform engine improvements tracked in gitlab-org/duo-workflow&1.

Edited by Mikołaj Wawrzyniak