example 7

Full Analysis of Merge Request gitlab-org/editor-extensions/gitlab-lsp!1323 (merged) and Related Work for Import-Based Code Suggestions Context

This report provides a comprehensive analysis of Merge Request gitlab-org/editor-extensions/gitlab-lsp!1323 and its closely related Merge Requests and issues. The central theme of this work is the enhancement of GitLab Duo Code Suggestions by incorporating "Imports Context" – context derived from files imported by the currently active document. This initiative aims to significantly improve the accuracy and relevance of AI-generated code suggestions.

Key Merge Request: gitlab-org/editor-extensions/gitlab-lsp!1323: chore: introduce JavascripttImportStore+

1. Overall Goal: Enhancing Code Suggestions with Import Context

The primary strategic goal behind this body of work is to improve the quality, accuracy, and relevance of GitLab Duo Code Suggestions. Current Code Suggestions primarily rely on the content of the open file and cursor position. This limited context often results in suggestions that lack awareness of related code, dependencies, and project structure, particularly in larger or more complex projects.

By analyzing import statements, the Language Server can identify and incorporate content from files that the current document explicitly depends on. This provides the AI model with a richer understanding of the surrounding codebase, leading to more intelligent and helpful suggestions. This initiative is a key part of broader efforts to advance context retrieval for AI tools within GitLab, as detailed in epics like:

As stated in the description of epic gitlab-org/editor-extensions&58 (closed)+:

Code Suggestions currently uses open files and cursor context to show suggestions. Adding import statement analysis will provide better context about related files and dependencies.

Issue gitlab-org/gitlab#503839: Research: Additional contexts and Import context+ further elaborates on the collaboration and planning for how Imports Context will be implemented and consumed.

2. Architectural Overview of the Import Context System

The implementation of the Import Context feature involves several key components within the GitLab Language Server (gitlab-lsp):

The architecture diagram from MR gitlab-org/editor-extensions/gitlab-lsp!1311 (closed)+ (Draft: feat: ImportContextProvider) illustrates this:

classDiagram
    direction TB

    class AIContextManager {
        +getContext(IDocContext)
        +getContextForCodeSuggestions(CodeSuggestionsAIRequest)
    }

    class DefaultImportContextProvider {
        -logger: Logger
        -fsClient: FsClient
        -importHandlers: AbstractImportHandler[]
        -projectAccessChecker: DuoProjectAccessChecker
        -policy: FilePolicyProvider
        -openTabsService: OpenTabsService
        +getContextForCodeSuggestions(request: CodeSuggestionsAIRequest)
    }

    class AbstractImportHandler {
        <<abstract>>
        +getSupportedLanguages()* : TreeSitterLanguageName[]
        +getTreeSitterQuery()* : Promise<string>
        +getImportPathsToMetadata()* : Promise<Map>
        +resolveImportPath()* : Promise<URI>
        +enabledForLanguage(languageName) : boolean
    }

    class EcmascriptImportHandler {
        -logger: Logger
        -importStore: ECMAScriptImportStore
        -fsClient: FsClient
        +getSupportedLanguages()
        +getTreeSitterQuery()
        +getImportPathsToMetadata()
        +resolveImportPath()
        -getRequireImportPath()
        -getAssociatedImportPath()
        -getDynamicImportPath()
    }

    class ECMAScriptImportStore {
        -filePathStore: Map
        -repositoryService: RepositoryService
        -configService: ConfigService
        +getFile()
        +storeReady()
        -addFile()
        -removeFile()
        -setupFileSystemListener()
    }

    AIContextManager --> DefaultImportContextProvider : uses
    DefaultImportContextProvider --> AbstractImportHandler : delegates to
    AbstractImportHandler <|-- EcmascriptImportHandler : implements
    EcmascriptImportHandler --> ECMAScriptImportStore : uses for lookup

3. Deep Dive into MR gitlab-org/editor-extensions/gitlab-lsp!1323: JavascriptImportStore

Merge Request gitlab-org/editor-extensions/gitlab-lsp!1323 (merged)+ introduces the JavascriptImportStore, a critical performance optimization for resolving ECMAScript (JavaScript, TypeScript, Vue) imports.

Purpose and Technical Contribution: The primary goal of the JavascriptImportStore is to provide fast, in-memory lookups for files relevant to JavaScript-based import resolution. This minimizes filesystem overhead, which can be significant, especially in large projects.

As stated in the MR description:

This Merge Request introduces a new JavascriptImportStore dedicated to fast file lookups for Javascript-based import resolution. It maintains an in-memory mapping of recognized files (e.g., .ts, .js, .vue) within a workspace folder so that import statements can be resolved with minimal filesystem overhead. This helps us better achieve our sub 50ms latency target.

Implementation Details:

  • DefaultJavascriptImportStore Class:

    • Maintains a nested Map (#filePathStore) where the outer key is the workspace folder URI and the inner map stores normalized file paths to their actual URIs.
    • Tracks files with specific extensions (e.g., .ts, .js, .vue) defined in TRACKED_EXTENSIONS.
    • Normalizes file paths by stripping common extensions to facilitate lookups (e.g., import './utils' can find utils.ts or utils.js).
    • Subscribes to RepositoryService events (onFileChange, onWorkspaceRepositoriesStart, onWorkspaceRepositoriesFinished) to keep its internal map synchronized with filesystem changes.
    • Provides a findFile method for looking up file URIs based on potential import paths.

  • Enhancements to Repository and RepositoryService:

    • Repository class: A getFiles method was added to retrieve files tracked by the repository, with options to exclude git-ignored or .git files.
    • RepositoryService class: Enhanced to broadcast file system changes and repository lifecycle events (onFileChange, onWorkspaceRepositoriesStart, onWorkspaceRepositoriesFinished). This allows components like JavascriptImportStore to react to changes incrementally without rescanning the entire project.

    The MR description further explains:

    Additionally, this MR extends the Repository and RepositoryService classes to better-broadcast file changes to interested consumers like the new JavascriptImportStore. By exposing getFiles() on Repository and emitting file updates via onFileChange, downstream components can efficiently track newly created, changed, or deleted files without rescanning the entire project.

Code Snippet: JavascriptImportStore Event Handling (Illustrative) From src/common/ai_context_management/context_providers/imports/ast/javascript/javascript_import_store.ts in MR !1323:

#setupFileSystemListener() {
  this.#repositoryService.onFileChange((event) => {
    const fileUri = parseURIString(event.fileEvent.uri);
    const { workspaceFolder } = event;
    switch (event.fileEvent.type) {
      case FileChangeType.Created:
      case FileChangeType.Changed:
        this.#addFile(fileUri, workspaceFolder);
        break;
      case FileChangeType.Deleted:
        this.#removeFile(fileUri, workspaceFolder);
        break;
      default:
        break;
    }
  });

  // ... (handling for onWorkspaceRepositoriesStart and onWorkspaceRepositoriesFinished)
}

Performance Impact: The JavascriptImportStore is crucial for meeting the strict performance targets (sub-50ms additional latency) for Code Suggestions. By caching file paths, it drastically reduces the time spent on filesystem I/O during import resolution.

Relation to ImportContextProvider: The JavascriptImportStore is a dependency of the JavascriptImportResolver (or EcmascriptImportResolver), which in turn is used by the ImportContextProvider. The resolver queries the store to quickly find the URIs of imported modules.

Comments on MR !1323: Review comments, such as those by elwyn-gitlab, focused on test coverage, path handling, and confirming the soundness of the in-memory map approach for performance.

  • elwyn-gitlab (2025-02-05T14:59:17Z):

    Looks good overall! This is a great step towards getting the import context provider working performantly. I have a few minor comments and questions below.

  • elwyn-gitlab (2025-02-05T15:03:43Z): (Suggesting a test case for extension handling) 
    // For TypeScr... (truncated)

    Could we add a test for this case? E.g. adding a .ts file and then checking that findFile works with and without the .ts extension. michaelangeloio (Author) (2025-02-05T15:06:56Z): Good idea, I'll add a test for this.

4. Closely Related Merge Requests and Their Contributions

The development of the Import Context feature is an "MR Train," with several MRs building upon each other:

  • gitlab-org/editor-extensions/gitlab-lsp!1321: chore: introduce AbstractImportHandler+

    • Contribution: Introduced the AbstractImportHandler (later AbstractImportResolver) class. This abstract class defines a standardized interface for language-specific import analysis, requiring implementations to provide Tree-sitter queries, extract metadata, and resolve import paths. It also introduced core types like ImportIdentifier, ImportMetadata, and ResolvedImportMetadata.
    • Significance: Established the foundational abstraction for extensible, language-specific import processing.

  • gitlab-org/editor-extensions/gitlab-lsp!1322: chore: introduce JavascriptImportResolver+

    • Contribution: Introduced the EcmascriptImportResolver (or JavascriptImportResolver), a concrete implementation of AbstractImportResolver for JavaScript, TypeScript, TSX, and Vue files. It uses Tree-sitter queries to identify various import syntaxes (ESM, CommonJS, dynamic imports) and extracts import paths and identifiers.
    • Significance: Provided the first language-specific implementation for parsing ECMAScript imports, a core requirement for the feature.
    • Code Snippet: Tree-sitter Query from query.ts (MR !1322) 
      export const treeSitterQuery = `(
  [
    ; 1. Named imports (with optional alias)
    (import_statement
      (import_clause
        (named_imports
          (import_specifier
            name: (identifier) @${captureMap.importedIdentifier}
            alias: (identifier)? @${captureMap.renamedIdentifier}
          )
        )
      )
      source: (string) @${captureMap.importSource}
    )
    // ... other import types (default, namespace, require, dynamic) ...
  ]
)`;

  • gitlab-org/editor-extensions/gitlab-lsp!1324: chore: add relative module support for JavascriptImportResolver+

    • Contribution: Enhanced the JavascriptImportResolver to correctly resolve relative import paths (e.g., ./file, ../dir, directory imports resolving to index files). It introduced resolveRelativeModule logic.
    • Significance: Made the resolver functional for a common and crucial type of import in JavaScript/TypeScript projects. It leverages the JavascriptImportStore for lookups.

  • gitlab-org/editor-extensions/gitlab-lsp!1311: Draft: feat: ImportContextProvider+ / gitlab-org/editor-extensions/gitlab-lsp!1325: feat: introduce ImportContextProvider+

    • Contribution: Introduced the DefaultImportContextProvider class. This class orchestrates the import analysis process by selecting the appropriate language handler (resolver), triggering AST parsing and import metadata extraction, resolving import paths, applying file policies, and creating ImportAIContextItems.
    • Significance: This is the main service that ties together the resolvers and stores to provide import-based context to the AIContextManager.
    • Quote from MR !1325 Description:

      This Merge Request debuts a brand-new ImportContextProvider, allowing our AI features—particularly Code Suggestions—to leverage file-import analysis from ECMAScript, TypeScript, and other languages (via resolvers) in a standardized way.

  • gitlab-org/editor-extensions/gitlab-lsp!1327: feat: introduce code suggestion context availability checking+

    • Contribution: Implemented the client-side logic for feature flagging Code Suggestions context providers. It updated DuoFeatureAccessService to query a new GraphQL field (currentUser { codeSuggestionsContexts }) for enabled contexts. AIContextProviders can declare a duoRequiredFeature, and AIContextManager filters providers based on this.
    • Significance: Provided the mechanism to control the rollout of the Imports Context feature via a feature flag.

  • gitlab-org/gitlab!178402: Introduce codeSuggestionsContexts field to CurrentUser GraphQL type+

    • Contribution: Added the codeSuggestionsContexts field to the CurrentUser GraphQL type on the GitLab backend. This field returns a list of enabled context categories (e.g., "imports") based on feature flag checks (e.g., code_suggestions_include_context_imports).
    • Significance: Provided the server-side support for the client-side feature flagging mechanism implemented in !1327.

  • gitlab-org/editor-extensions/gitlab-lsp!1237: Draft: chore: expand AIContextManager to accept IDocContext+ (Closed Draft)

    • Contribution (Intended): Aimed to refactor AIContextManager and related components to accept IDocContext, allowing providers to access current document information.
    • Significance: Showed the evolution towards more flexible context providers that are document-aware, a prerequisite for import analysis.

  • gitlab-org/editor-extensions/gitlab-lsp!1251: test: add integration tests for context filters+

    • Contribution: Added comprehensive integration tests for advanced context filtering logic (byte size limits, project access, supported languages, empty content).
    • Significance: Ensured the reliability of the context pipeline's filtering stage, which is crucial before and after refactoring context management.

5. Performance Considerations

Performance is a critical requirement for the Import Context feature, with a target of sub-50ms additional latency for Code Suggestions requests when imports context is enabled (gitlab-org/editor-extensions/gitlab-lsp#775: Imports Context - Initial Iteration - Rollout Plan+).

Key strategies and components addressing performance:

6. Feature Flagging and Rollout Strategy

The Imports Context feature is rolled out using a feature flag to allow for controlled testing and gradual release.

  • Backend Feature Flag: code_suggestions_include_context_imports is defined in the GitLab Rails application.
  • GraphQL Exposure: MR gitlab-org/gitlab!178402 (merged)+ exposes the status of this flag (and others) via the currentUser { codeSuggestionsContexts } GraphQL field.
  • Client-Side Check: MR gitlab-org/editor-extensions/gitlab-lsp!1327 (merged)+ updates the DuoFeatureAccessService in the LSP to query this GraphQL field. The AIContextManager then uses this service to determine if the ImportContextProvider should be active.
  • Rollout Plan: Issue gitlab-org/editor-extensions/gitlab-lsp#775+ outlines the rollout strategy, success criteria (quality improvement, latency targets), and feedback collection.

This multi-layered approach ensures that the feature can be enabled or disabled at the user level, managed from the backend.

7. Integration into the Code Suggestions Context Pipeline

The Imports Context is integrated into the existing Code Suggestions context pipeline as follows:

  1. The AIContextManager queries all registered context providers.
  2. If the ImportContextProvider is enabled (via feature flag check in !1327), it analyzes the current document's imports using its resolver (e.g., JavascriptImportResolver from !1322, !1324) and the JavascriptImportStore (!1323).
  3. The ImportContextProvider returns AIContextItems representing the resolved imported files.
  4. These items, along with context from other providers (e.g., Open Tabs), are passed to a pre-processor pipeline.
  5. The pipeline filters items (tested in !1251), ranks them using a ContextRanker (gitlab-org/editor-extensions/gitlab-lsp#698 (closed)+), and retrieves content for the top-ranked items.
  6. The final, processed context is included in the payload sent to the Code Suggestions API.

8. Conclusion

Merge Request gitlab-org/editor-extensions/gitlab-lsp!1323 is a pivotal piece in a larger, coordinated effort to enhance GitLab Duo Code Suggestions by incorporating context from imported files. The JavascriptImportStore it introduces is a critical performance optimization, enabling fast resolution of ECMAScript imports. This store is a foundational component for the ImportContextProvider and its associated language-specific resolvers.

The related MRs and issues demonstrate a comprehensive approach, covering:

  • Architectural Design: Abstracting language-specific logic (!1321), implementing concrete resolvers (!1322, !1324), and orchestrating context gathering (!1311, !1325).
  • Performance: Prioritizing low latency through in-memory stores and optimized analysis strategies (#710, !1323).
  • Feature Control: Implementing robust feature flagging on both backend (gitlab!178402) and client (!1327) for controlled rollout (#726).
  • Pipeline Integration: Ensuring the new context source fits seamlessly into the existing context processing pipeline, including filtering and ranking.

This body of work represents a significant advancement in providing richer, more relevant context to AI models, with the ultimate aim of delivering more accurate and helpful Code Suggestions to GitLab users. MR !1323 plays a vital role by ensuring that this enhanced context can be gathered efficiently.