AI Catalog support for Self Managed & Dedicated

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

This issue was started to track the conversation started in #549401 (comment 2560617832) around how we:

  • Have catalog support for Self Managed and Dedicated instances
  • Are fully Cells compatible

Support for Duo Agent Platform features is planned for Self Managed and Dedicated for 18.2, so our catalog must work with these environments post-experiment stage.

Note these are 2 separate concerns but the solution may be related and so for now we are discussing them in one issue. We may want to split this into two separate issues at some point.

Database Architecture & Cells

  • The team discovered that "global/instance-wide" data no longer exists in GitLab's architecture due to the Cells initiative
  • Organizations and cells are related but separate concepts - multiple organizations can exist within a single cell
  • Data must now be stored at the organization level (gitlab_main_org) rather than globally

Technical Challenges Identified

  • Cross-organization querying: How to search/filter across multiple organizations when they're in separate cells
  • Data synchronization: How to ensure new organizations get access to system catalog items
  • Publishing mechanism: How users can publish their items to make them available to others

Proposal

Summarised here for organizations/cells and here for SM and Dedicated

The summary of that approach is: We can have them populate their catalogs via an API call to download AI Catalog data (that the token is permitted to see) from another organisation’s catalog. The feature is summarised under the “what do we need to do in future?” comment heading here. We can allow filtering options - this is where “GitLab official” #557843 (closed) would be great.

Click to read potential solutions discussed

Potential Solutions Discussed

Self-Managed / Dedicated support

See thread #549767 (comment 2580351934).

  • Static data: Store system-defined items as static data, for example migrations or YAML files
  • API syncing: Allow instances to pull catalog items from another instance (example, GitLab.com)

Cells support

  • Microservice approach: Build a separate service (though this adds complexity)
  • Single organization approach: Store the catalog on a single organization (either the default or a dedicated org). Other organizations make requests to that one organization. In Cells 2.0 this could work, though there are no public organizations in Cells 1.
  • YAML-backed models: Store system-defined items as static YAML files rather than in the database

Rejected Solutions

  • Elasticsearch approach: Use ES to maintain a searchable index across all cells for catalog items - not cells compatible #549767 (comment 2580306531)
Edited by Thong Kuah