## Definition Duo Self-Hosted Platformization is an initiative to enable teams to fully own their AI features (development and evolution) on Duo Self-Hosted. The goal is to give feature teams the tools they need to own and drive development of their AI-powered features in self-hosted environments, closing the gap between availability of Duo features on SaaS, Dedicated, Self-Managed, and Self-Hosted. A key goal of this initiative is to make Self-Hosted feature development and maintenance as seamless as possible, with minimal extra work required to bring a feature to Self-Hosted. Self-Hosted Platformization will align with and extend the vision under development by AI Framework for [AI feature development](https://docs.google.com/document/d/1mCAJ3goOQCfSm2Ho7tmJ_j-KyJAnYlLqWIBlj3q8xYM/edit?tab=t.0#heading=h.l6lixya3upkb) and [model validation operationalization](https://gitlab.com/groups/gitlab-org/-/epics/16605#note_2392303949). This involves: * Extending AIF platforms to support Self-Hosted specific use cases, to include model provisioning * Creating a collection of tools, services, and documentation that enable feature teams to self-serve on the development and maintenance of Self-Hosted versions of their features * Shifting ownership of AI feature maintenance and development from the Custom Models team to the respective feature teams * Decentralizing responsibility so that all self-hosted AI features aren't managed solely by the Custom Models team ## Problems It Will Solve 1. **Inconsistency Between Environments**: Currently, there are discrepancies between SaaS, Self-Managed, Dedicated, and Duo Self-Hosted feature implementations. This platformization will provide resources and tools to easily bring consistency across supported environments. 2. **Ownership and Accountability Issues**: The current setup causes unexpected conflicts and errors because teams don't fully own their AI features in self-hosted setups. Changes made to features on SaaS may not be carried over to Self-Hosted versions, and Custom Models may not have awareness of those changes -- leading to misalignment and bugs. 3. **Domain Knowledge Gap**: The Custom Models team lacks specialized domain knowledge about specific features (e.g., Duo Code Review, Duo Chat), putting them in a poor position to make decisions about how these should behave in self-hosted environments. 4. **Scalability Issues**: As Duo features evolve and new features are released, it's becoming untenable for the Custom Models team to maintain awareness of all changes, adapt features to work with self-hosted options, and test each feature change within each milestone. 5. **Support Burden**: The Custom Models team cannot sustainably provide customer support for each self-hosted version of a feature as the number of features grows. ## Core Responsibilities In order to address the above issues, a clear division of responsibilities is required to ensure that self-hosted capabilities are developed in sync with AI features. This plan ensures that self-hosted Duo feature releases and evolutions will be scalable and sustainable across different environments. The below represents the proposed scopes of responsibilities: ### AI Framework Team: * **Development Patterns**: * Define AI feature development patterns and workflows * Own prompt versioning * Be opinionated about how teams develop AI features * **Evaluation Tools**: * Consolidate Evaluation Runner * Add new datasets to Evaluation Runner * add new evaluators to Evaluation Runner * Own SSOT repository * Provide methodology for creating subsets of data ### Custom Models Team: * **Platform and Infrastructure**: * Provision and sustain [models supported by GitLab Duo Self-Hosted](https://docs.gitlab.com/administration/gitlab_duo_self_hosted/supported_models_and_hardware_requirements/) for use in feature development and evaluation * Maintain AI settings and configuration patterns, to include self-hosted configurations * Vet and maintain a set of core models * Introduce and deprecate models as necessary * Maintain Self-Hosted platform and documentation * Continue to iterate on and introduce new tools to further streamline Self-Hosted development (i.e. automated prompt optimization) * **Developer Experience**: * Document processes for developing Self-Hosted features from the ground up * Give feature teams easy access to Self-Hosted models for development and validation * Enable automated evaluations for features across different models * Train feature teams on supporting self-hosted models in both cloud and air-gapped environments ### Feature Teams: * **Feature Development and Maintenance**: * Own prompts for self-hosted models * Define what it means for the feature to work well * develop and own SSOT validation datasets for their features * Perform evaluation against self-hosted models * Maintain the self-hosted version of their features * **Feature Updates and Quality**: * Update prompts and related code when making upstream changes to features * Ensure changes made to features don't break models introduced by Custom Models * Update self-hosted version features to keep them aligned in terms of functionality and quality * Provide customer support for the self-hosted version of their feature * **New Feature Development**: * Develop and own self-hosted versions of new features * Ensure self-hosted considerations are baked in from the start in feature development ### Code Activities: * **Consolidate Evaluation Runner** (AI Framework) * Ensure all datasets are available on Evaluation Runner * Improve reporting capabilities in Evaluation Runner * Implement CI-driven evaluation (triggered from MR requests) * **Model Provisioning** (Custom Models) * Extending Eval Runner for Self-Hosted use * Adding new models to Evaluation Runner * Make models available for Gitlab developer usage * Allow any AIGW endpoint to support self-hosted models ### Documentation Activities: * **Feature Development Documentation** (AI Framework) * How to develop a GitLab Duo feature * how to evaluate using AI Framework's Evaluation Runner * How to interpret evaluation scores * **Self-Hosted Feature Development** * How to add a new feature (configuration lists, compatible models) * How to evaluate across Duo Self-Hosted models using Evaluation Runner * How to use a provisioned model * **Prompt Engineering Guidelines**: * How to write prompts for different model families * Best practices for creating prompts that can be adapted to other models * How to run evaluations on evaluation-runner for Self-Hosted models ## References * discussion issue for team alignment https://gitlab.com/gitlab-org/gitlab/-/issues/525035+ * [AIF/CM Platform Sync](https://docs.google.com/document/d/1u8ehK4ydZ9oWAcILc0KncKyLpOjKvJZncJcpwD1tnFk/edit?tab=t.0) * [CC AI Platform Interface Vision](https://gitlab.com/gitlab-org/ai-powered/discussions/-/issues/28) / [CC Technical Vision](https://handbook.gitlab.com/handbook/engineering/infrastructure/team/cloud-connector/technical_vision/) * [AIF GL Model Validation Operationalization](https://gitlab.com/groups/gitlab-org/-/epics/16605#note_2392303949) * [AIF Model Eval Bridge Process](https://gitlab.com/groups/gitlab-org/-/epics/16871) * [Handbook page for AI Model Validation](https://handbook.gitlab.com/handbook/engineering/development/data-science/ai-powered/model-validation/model_evaluation/) * [AI Feature Development Playbook](https://docs.gitlab.com/development/ai_features/ai_feature_development_playbook/) ---