WIP - Context and embeddings abstraction layer : Product requirements

Note: This issue is meant to pair with the Context and embeddings layer Blueprint MR, providing product requirements to map the technical requirements to.

Problems to solve

The current approach to context creation and embeddings at GitLab lacks the flexibility and adaptability needed to support diverse AI-driven features across different domains. Specifically:

  1. Domain teams need to be able to easily customize indexing and embedding processes for their specific use cases.
  2. Creating and managing embeddings should be simple for domain teams, requiring no specialized knowledge.
  3. Retrieving embeddings should be straightforward and multiple approaches should be supported (e.g., hybrid search)
  4. Content processing for embeddings should be flexible, providing options for what and how to index.
  5. Provide support for creating and managing rich meta-information, especially for repositories.
  6. Support long-term context management for user interactions and workflows.

Product requirements

The below items are in rough priority and subject to change based on stakeholder feedback. We will split these items into multiple iterations. We are targeting 2024-10-04 for an initial proposed iteration plan.

  1. Abstraction Layer
    • Provide a storage-agnostic abstraction layer for embedding creation, storage, and retrieval
    • Support multiple backend storage solutions (e.g., PGVector, Elasticsearch)
    • Allow easy switching between storage solutions without major code changes
    • Provide a comprehensive API for other GitLab services to interact with the context and embeddings layer
    • Ensure backwards compatibility and provide clear migration paths for existing AI features
  2. Performance and Scalability
    • Ensure low-latency retrieval of embeddings and context for real-time applications
    • Support efficient bulk operations for indexing and updating large volumes of content
    • Implement caching mechanisms to optimize frequent queries
  3. New Context Generation
    • Support the creation of new meta-context information using AI models
    • Enable the generation and storage of pre-computed questions or summaries for content
    • Facilitate the integration of AI-generated insights into the context layer
  4. Customization for Domain Teams
    • Enable domain teams to define custom indexing strategies for their specific content types
    • Provide intuitive interfaces for teams to configure embedding parameters without deep AI expertise
    • Allow teams to specify what content should be indexed and how it should be combined
  5. Flexible Content Processing
    • Implement configurable chunking strategies suitable for various content types (e.g., code, documentation, issues)
    • Support customizable pre-processing and post-processing steps for indexed content
    • Allow selection of different embedding models based on content type or use case
  6. Meta-information Management
    • Facilitate the creation and storage of additional meta-information (e.g., file relationships, LLM-generated summaries)
    • Support the building and querying of knowledge graphs
    • Enable easy integration of linting warnings and other automated analyses into the context
  7. Monitoring and Management
    • Implement logging and monitoring solutions to track usage, performance, and errors
    • Provide tools for managing and updating embeddings, including reindexing capabilities
    • Create interfaces for viewing and managing the context and meta-information associated with content
Edited by Ben Venker