Dedicated Instance Testing for DAP Features and Usage Billing

Background

This issue originates from the corrective actions identified in the RCA for GitLab Duo Chat Disruption (INC-6757).

During the incident, Dedicated customers on older versions (18.6.3-internal2, 18.7.1_internal2) encountered a "No GitLab Credits Remain" error when using Duo Agent Platform (DAP). The root cause was a policy change that required user information during workflow token generation, which older Dedicated instances did not send, causing token requests to fail with a quota error.

Problem Statement

We need comprehensive documentation and processes for testing DAP features and usage billing functionality on Dedicated instances to prevent similar compatibility issues in the future.

Objectives

Create documentation that addresses the following:

1. Supported Dedicated Instances for DAP Features

   • Document which Dedicated instance versions support DAP features
   • Define the minimum supported version for DAP (e.g., is it 18.6, 18.8, or another version?)
   • Maintain a version compatibility matrix for DAP features

2. Testing Requirements for Usage Billing Features

   • Identify what changes in usage billing features require Dedicated instance testing
   • Identify what additions of new features require Dedicated instance testing
   • Define testing criteria and acceptance requirements

3. Backward Compatibility Considerations

   • Document backward compatibility requirements when adding new usage billing features
   • Document backward compatibility requirements when changing existing usage billing features
   • Establish guidelines for policy changes that affect token generation or quota validation

4. Testing Process and Procedures

   • Document how usage billing features should be tested on Dedicated instances
   • Define test scenarios and coverage requirements
   • Establish testing timelines relative to feature releases

5. Future Iteration: Test Automation

   • Explore and document ways to automate Dedicated instance testing for usage billing features
   • Identify opportunities for continuous validation across supported versions

Success Criteria

• Documentation clearly identifies which Dedicated instances support DAP features
• Clear guidelines exist for when Dedicated instance testing is required
• Backward compatibility considerations are documented and enforced
• Testing procedures are documented and can be followed by engineering teams
• Automation opportunities are identified for future implementation

Related Issues

• RCA: GitLab Duo Chat Disruption (#21143)
• Process Gap: No mechanism to prevent known Self-Managed bugs from impacting Dedicated deployments (#588460)

Additional Context

From the incident:

• The policy change was deployed without considering backward compatibility with older Dedicated versions
• No documented validation across Dedicated versions existed before the policy change
• No version matrix testing was documented for workflow token generation changes