Related issue: https://gitlab.com/gitlab-org/gitlab/-/issues/465221 # DRI * @nmilojevic1 # Participants - @alipniagov - @mkaeppler - @rzwambag ### **Epic: Extract CloudConnector Unit_Primitive Configuration and Logic** **Summary:** This epic addresses the need to migrate from the existing service-based configuration to a new unit primitive-based configuration for CloudConnector. The current configuration, used for both self-managed and GitLab.com customers, lacks flexibility and is hard to extend when adding new metadata or controlling access to unit primitives. The unit primitive configuration will be extracted to the [`gitlab_cloud_connector`](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector) project, serving as the single source of truth (SSoT) for all unit primitive definitions. This project will house the configuration YAML files, as well as both a Ruby gem and a Python package, enabling all backend systems—including GitLab Rails, CustomersDot, Ai Gateway, and Duo Workflows—to access the configuration consistently. Additionally, the `CloudConnector::AvailableServices` interface will be revisited to resolve confusion caused by its service abstraction and refine token management for both GitLab.com and Self-Managed environments. The migration must maintain backward compatibility with older client versions, prevent disruption in historical reporting, and provide a clear path for clients and internal systems to transition smoothly. --- ### **Problem Statement:** 1. **No SSoT for CloudConnector Configuration:** - The current system relies on distinct configurations for Self-Managed (CustomersDot) and GitLab.com (access_data.yml), leading to challenges in maintaining consistency across instances. - The AI Gateway and Duo Workflow also have configuration mappings related to unit primitives. A unified, central approach is needed for unit primitives to be consistent across all environments. 2. **Existing Structure Lacks Flexibility:** - Difficult to control cut-off dates, version constraints (`min-gitlab-version`), and add metadata for each unit primitive. - The current configuration structure does not allow for storing more useful metadata per Unit Primitive, such as: `description`, `documentation_link`, `feature_category`, `introduced_by_url` - The lack of clear associations between unit primitives, add-ons, and license types makes it hard to understand the relationships at a glance. 3. **Misalignment of CloudConnector::AvailableServices Interface:** - The `AvailableServices` interface is organized around the concept of a "service," which represents a UI component or how unit primitives are accessed, rather than the unit primitive itself. - This abstraction introduces confusion because it conflates UI and access control, leading to unnecessary complexity when determining feature access based on user permissions and seat assignments. 4. **Separation of Concerns:** - The current use of `AvailableServices` for end-user permission checks, such as verifying add-on purchases or seat assignments, introduces UI-focused logic into CloudConnector. This should be moved to a different domain, leaving CloudConnector to focus solely on providing unit primitive data and relevant access details. 5. **Token Handling and Discrepancies Between GitLab.com and Self-Managed Instances:** - The current system creates a gap between GitLab.com and Self-Managed instances: - For Self-Managed, a JWT instance token is issued by CustomersDot, containing all unit primitives to which the instance has access. - For GitLab.com, a JWT instance token is issued per service, and the scopes included are based on end-user permissions (seat assignments) - The goal is to reduce this gap by aligning both environments with a consistent token issuance process. This would allow CloudConnector to issue instance tokens across both environments, ensuring secure and consistent access control. --- ### Goals - Implement the new unit primitive-based configuration. - Maintain backward compatibility with older client versions. - Ensure historical data remains accurate for reporting purposes. - Provide a clear migration path and communication plan for clients and internal systems. --- ### **Proposed Solution:** To address the challenges outlined in the problem statement, we propose the following solution: 1. **Adopt a Unit Primitive-Based Configuration:** - **Create Individual YAML Files for Each Unit Primitive:** - Store each unit primitive in its own YAML file within the `config/unit_primitives/` directory. - Include metadata such as `cut_off_date`, `min_gitlab_version`, `add_ons`, `license_types`, `description`, `documentation_url`, etc. - This modular approach simplifies updates and enhances maintainability. 2. **Develop a Unified Configuration Library:** - **Introduce a `gitlab-cloud-connector` Ruby Gem and Python Package:** - Encapsulate the configuration logic in a Ruby gem and a Python package. - Serve as the single source of truth (SSoT) for unit primitives across all systems: GitLab Rails, CustomersDot, AI Gateway, and Duo Workflows. - Provide a consistent API for accessing unit primitive data. 3. **Refactor the `AvailableServices` Interface:** - **Shift Focus from Services to Unit Primitives:** - Replace the `AvailableServices` abstraction with direct access to unit primitives. - Simplify the interface by removing UI-related permission checks. - Allow CloudConnector to focus solely on feature-level data. 4. **Standardize JWT Token Issuance:** - **Extract Token Issuing Logic into the Shared Library:** - Move JWT token generation into the `gitlab-cloud-connector` gem/package. - Ensure consistent token handling across GitLab.com and self-managed instances. - Generate instance tokens containing the necessary scopes based on unit primitives. 5. **Maintain Backward Compatibility:** - **Implement Server-Side Mapping for Legacy Support:** - Map unit primitives back to services for older clients. - Modify the GraphQL resolver to reconstruct the `availableServices` field. - Deprecate `availableServices` in the GraphQL schema but keep it available until clients have migrated. 6. **Update Client Applications and Reporting Systems:** - **Clients:** - Update clients on how to use the new unit primitive-based configuration. - **Reporting Systems:** - Adjust reporting logic to map both old and new unit primitives to features. - **Ensure historical data remains accurate by maintaining mappings for deprecated unit primitives.** - Requires further investigation --- ### Phases To ensure a smooth migration while retaining all necessary details, we will proceed through the following phases, incorporating the detailed next steps: #### Phase 1 - Implement new Unit Primitive configuration 1. **Update Configuration in `gitlab-cloud-connector` Project** - **Add New Unit Primitive YAML Configuration:** - [x] Create individual YAML configuration files for each unit primitive in `config/unit_primitives/`. - Ensure these files contain the necessary metadata, such as `cut_off_date`, `min_gitlab_version`, `add_ons`, `license_types`, `feature_category`, `description`, `documentation_url`. - **Set Up CI/CD Tools for Tests, Linting, and Validation:** - Implement a CI/CD pipeline that runs the following checks: - [x] **Linting:** Ensure the YAML and tooling code follow best practices. - [x] **Schema Validation:** Validate the unit primitive YAML configuration using JSON schema. - [x] **Tests:** Run RSpec for the tools to ensure the gem behaves as expected. - **Documentation:** - [x] Provide instructions for adding and updating unit primitives using the new tools. - [ ] Document the CI/CD processes for testing, linting, and validation. ### Phase 2: Deliver and Publish the Ruby Gem for Configuration Access 1. **Create and publish the `gitlab-cloud-connector` Ruby gem and Python package.** - **Ruby Gem:** - [x] Add Rails gem to `gitlab-cloud-connector` project. - [x] Set up the publication process via RubyGems. - [x] **Set Up CI/CD Tools:** - [x] Implement linting (e.g., RuboCop) and testing (RSpec) for the Ruby gem. - [x] Use `gem-release` for automatic publishing. 2. **Create ADR for Unit Primitives configuration structure** 3. **Extract JWT Access Token** * **Ruby Library:** * [x] Build a Ruby library that can issue JWT Access Token. This should be configurable, so we can configure subject, the issuer, and the TTL 4. **Use Ruby Gem as SSoT to access Unit Primitive Configuration** - **Ruby Library:** - [x] \[**In Progress**\]Build a Ruby library that can read the new unit primitive configuration. - [x] Create adapter to convert new unit primitives to old "services" yml configuration - **CustomersDot:** - [x] Replace `cloud_connector.yml` with the Ruby gem's configuration access. - [x] Use the adapter to maintain compatibility with existing interfaces. - **Gitlab.com:** - [x] Replace `access_data.yml` with the Ruby gem's configuration access. - [x] Use the adapter to maintain compatibility with existing interfaces. 5. **Automate Publishing and Version Bumping Process** - **Ruby Gem:** - [x] Automate the publishing process using `gem-release`. - [x] Establish a process for bumping gem versions in dependent systems (e.g., GitLab Rails, CustomersDot). - Create templates similar to what exists for `gitlab-styles` for merge requests when updating versions. --- ### Phase 3: \[Draft\] Unify Token Issuance 1. [ ] **Unify Token Issuance Logic Across Platforms** - While we did extract JWT Access Token into the library in Phase 2, the token issuance logic is inconsistent: - **GitLab.com:** Issues tokens scoped to specific services (like `duo_chat`), based on individual user permissions such as seat assignments. This approach provides fine-grained access control. - **Self-Managed Instances:** Use a single, instance-wide token that includes all available unit primitives, regardless of individual user permissions. This token grants broad access to all enabled features. - **Solution:** Align token issuance with self-managed instances 2. [ ] **Extract Token Issuance logic to Ruby Gem.** 3. [ ] **Replace token issuance logic in CustomersDot and Gitlab.com.** 4. [ ] **Get rid of special cases for issuing tokens like `anthoripic-proxy`, `vertex-ai-proxy` and `self-hosted-models`** ### Phase 4: \[Draft\] Support synchronizing both new and legacy Unit primitive format For Self Managed instances we synchronize availableService data from CustomersDot. For older instances we should continue supporting the legacy format. But we should add support for synchronizing new format. 1. [ ] Modify the GraphQL schema to include both `availableServices` (deprecated) and `availableUnitPrimitives`. - Continue using the adapter to map new unit primitives to legacy services. Ensure that old clients continue to function without disruption. 2. [ ] Update documentation to reflect the deprecation. 3. [ ] Update GraphqlQuery to fetch 'avaiable_unit_primitives' instead of 'available_services' 4. [ ] Store `availableUnitPrimitives` to `CloudConnector::Access` data-table ### Phase 5: \[Draft\]\[Rails monolith\] Refactor AvailableServices interface 1. **Replace `AvailableService` Interface in Rails monolith** - [ ] **GitLab.com:** - use default configuration from `gitlab-cloud-connector` library - [ ] **Self-Managed:** configure `gitlab-cloud_connector` library to read data from `CloudConnector::Access` data-table (`availableUnitPrimitives`) - [ ] UI end-user permission checks, like seat-assigments, add-on purchases should be extracted in the authorization layer above the CC interface (e.g Ai::UserAuthorizable) - [ ] Handle special cases like `self-hosted-models` ## Additional improvements Outside of the scope of this work there are further improvements identified. They are tracked in [**Additional improvements to new Unit Primitive set up**](https://gitlab.com/groups/gitlab-org/-/epics/15954) ### Status 2025-02-26 * To improve the reporting, we have moved Phase 3,4,5 Epics out of this parent Epic and will be reporting on their progress individually. Future status updates will be provided directly on those Epics. ### Status 2025-02-12 * [Phase 2](https://gitlab.com/groups/gitlab-org/-/epics/15636) is complete! All that remains is to remove the feature flag. This means that we are now using our new configuration as the SSOT for Unit Primitive configuration. However under the hood, this is converted back to the "old" format for use across Self Managed and .com. * We continue with [Phase 4](https://gitlab.com/groups/gitlab-org/-/epics/15638#note_2314006902), which will move us towards actually using the new UP config format throughout the stack. ### Status 2025-02-05 * Roll out of https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/11268+ is now at 100% behind a feature flag. We will leave it like this until Monday and then remove the feature flag. Once complete this will bring[ Phase 2 ](https://gitlab.com/groups/gitlab-org/-/epics/15636)to completion. * We have also begun to build out the plan for [Phase 4](https://gitlab.com/groups/gitlab-org/-/epics/15638#note_2314006902) and have started working on the first items to support synchronising the new Unit Primitives format. This will likely be delivered in %"17.10". * For [Phase 3, ](https://gitlab.com/groups/gitlab-org/-/epics/15637)we will begin with a PoC that encompasses changes needed across [Phase 3](https://gitlab.com/groups/gitlab-org/-/epics/15637) and [Phase 5](https://gitlab.com/groups/gitlab-org/-/epics/15639). We have parked this work this week due to other priorities. ### Status 2025-01-29 * https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/11268+ has been merged behind a feature flag. We plan to complete the roll out<span dir=""> </span>via the feature flag over the next few days. Once complete this will bring[ Phase 2 ](https://gitlab.com/groups/gitlab-org/-/epics/15636)to completion. * As mentioned last week, for [Phase 3, ](https://gitlab.com/groups/gitlab-org/-/epics/15637)we will begin with a PoC this week that encompasses changes needed across [Phase 3](https://gitlab.com/groups/gitlab-org/-/epics/15637) and [Phase 5](https://gitlab.com/groups/gitlab-org/-/epics/15639). We have an issue for the [PoC](https://gitlab.com/gitlab-org/gitlab/-/issues/515279) and are beginning work on it now. * We have also begun to build out the plan for [Phase 4](https://gitlab.com/groups/gitlab-org/-/epics/15638#note_2314006902), with help from ~"group::provision" and are creating issues to allow us to begin implementation. ### Status 2025-01-22 * We expect https://gitlab.com/groups/gitlab-org/-/epics/15949+ to be completed by next week, which will bring[ Phase 2 ](https://gitlab.com/groups/gitlab-org/-/epics/15636)to completion. * For [Phase 3, ](https://gitlab.com/groups/gitlab-org/-/epics/15637)we will begin with a PoC this week that encompasses changes needed across [Phase 3](https://gitlab.com/groups/gitlab-org/-/epics/15637) and [Phase 5](https://gitlab.com/groups/gitlab-org/-/epics/15639). Once complete, we will have more clarity in the approach we want to take for these phases and can refine our plans. ### Status 2025-01-15 * We have broken up the work required in order to allow CDot to start using the new Unit Primitive configuration into multiple issues under https://gitlab.com/groups/gitlab-org/-/epics/15949 and are actively working on them. * To start to progress [Phase 3](https://gitlab.com/groups/gitlab-org/-/epics/15637), we are working on refining our approach to improve our Unit Primitive support for Custom Models. This will allow us to move forward with this phase, where we unify the token issuing logic, which is currently different for .com and Self Managed. ### Status 2025-01-08 * We have reached the last steps of [Phase 2](https://gitlab.com/groups/gitlab-org/-/epics/15636), which is to start using the new Unit Primitive config in CDot. We have an MR [in review](https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/11268) but it will be broken up into smaller chunks before being merged. * Updated Unit Primitive documentation has been [merged](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/175414#top). * New [PoC](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector/-/merge_requests/81) to generate a browsable Unit Primitive [catalog](https://gitlab-cloud-connector-nmilojevic1-fbdb8f6670974fcd8dd62f3fe91b.gitlab.io/) has been created. * Following this, in 17.9, we will begin [Phase 3](https://gitlab.com/groups/gitlab-org/-/epics/15637). ### Status 2024-12-18 * Completed roll out of usage of the new [Unit Primitive config](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector) to gitlab.com. The feature flag used for this roll out has now been removed. * Corresponding documentation updates are [in review](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/175414#top). * https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector/-/merge_requests/79+ has been merged. This generates a copy of the Unit Primitive configuration in the "old" format from the new one. This will be consumed by the analytics team and can be used until they migrate their code to consume the new Unit Primitive config format (being discussed [here](https://gitlab.com/gitlab-data/analytics/-/issues/22464#note_2261830143)). * Supporting analytics in this way unblocks us from the next steps, which is a similar process for [CDot](https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/11268), to start using the new Unit Primitive config. Once completed we will have a single source of truth for Unit Primitives via our Ruby library. ### Status 2024-12-11 * Started to roll out the use of the new [Unit Primitive config](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector) to gitlab.com behind a feature flag: https://gitlab.com/gitlab-org/gitlab/-/issues/507518+. This is now at 100%, and we will leave it like this for a few days before beginning the process of removing the old [config file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/cloud_connector/access_data.yml) for gitlab.com completely. * Corresponding documentation updates are [in review](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/175414#top) * Next up will be a similar process for the [CDot config file](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/config/cloud_connector.yml), moving over to use the new [Unit Primitive config](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector). ### Status 2024-12-04 * https://gitlab.com/groups/gitlab-org/cloud-connector/-/epics/2+ is now complete. We have extracted common Ruby code related to Cloud Connector JWTs into a Ruby library and integrated the library with CDot and GitLab. We also addressed a [performance issue ](https://gitlab.com/gitlab-org/gitlab/-/issues/506002)during this work related to loading the JWT signing key. * The [parser](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector/-/issues/10) and [adaptor](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector/-/issues/31) for the new Unit Primitive format are merged. We are now working on integrating this with the existing code, [starting with CDot](https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/11268). This work is on track to be completed in 17.7. Once complete, we will have a new [Unit Primitive SSOT](https://gitlab.com/groups/gitlab-org/-/epics/15625) config file. * We met with the Custom Models team and established next steps to [collaborate](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector/-/issues/21#note_2238680587) on cross cutting issues, including improving the way Unit Primitives are used for Custom Models. ### Status 2024-11-27 * We are continuing to work on https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector/-/issues/30+ to extract common Ruby code related to JWTs into a Ruby library to be shared between CDot and GitLab. This work is on track to be completed in 17.7 * We are continuing to work on a [parser](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector/-/issues/10) and [adaptor](https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector/-/issues/31) (for backwards compatibility) for the new Unit Primitive format. This work is on track to be completed in 17.7. Once complete, we can begin to migrate to the new [Unit Primitive SSOT](https://gitlab.com/groups/gitlab-org/-/epics/15625). ### Status 2024-11-20 * ADR outlining the changes we will make to the Unit Primitive system is [merged](https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/9791#top). * Continued working on https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector/-/issues/30+ to refactor and move JWT code to a shared Ruby library. * The work to refactor the JWT code in GitLab is [merged and rolled out](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/172378) behind a feature flag. Performance issue has been [found](https://gitlab.com/gitlab-org/gitlab/-/issues/503739#note_2221859452) and is being addressed. * Unit Primitive changes have been synced from the current config files to the new ones, allowing for a smoother migration to the new format when we are ready. We are working on https://gitlab.com/gitlab-org/cloud-connector/gitlab-cloud-connector/-/issues/10+ to provide a way to read the new config files and adapt them to the existing format as the first step on making the new config files be the SSOT for Unit Primitive configuration. ---