Documentation - Deep Research Agent
<!--IssueSummary start-->
<details>
<summary>
Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards.
</summary>
- [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=547270)
</details>
<!--IssueSummary end-->
This document provides a detailed overview of the https://gitlab.com/groups/gitlab-org/-/epics/17948+, its architecture, and the execution flow of its research stages. It's designed for developers and contributors who want to understand the inner workings of the agent.
**Table of Contents**
1. [Overview](#overview)
2. [Core Architecture](#core-architecture)
* [Knowledge Graph Integration](#knowledge-graph-integration)
* [Agentic Framework (LangGraph)](#agentic-framework-langgraph)
* [Orchestrator and Server Item Agents](#orchestrator-and-server-item-agents)
3. [Overall Execution Flow](#overall-execution-flow)
* [Mermaid Diagram: Top-Down Architecture](#mermaid-diagram-top-down-architecture)
4. [Detailed Execution Stages](#detailed-execution-stages)
* [Stage 1: Input Analysis & Item ID Extraction](#stage-1-input-analysis--item-id-extraction)
* [Reliable GitLab Item ID Extraction](#reliable-gitlab-item-id-extraction)
* [Stage 2: Knowledge Graph Indexing (Optional)](#stage-2-knowledge-graph-indexing-optional)
* [Mermaid Diagram: Graph Indexing](#mermaid-diagram-graph-indexing)
* [Stage 3: Research Plan Construction](#stage-3-research-plan-construction)
* [Mermaid Diagram: Plan Construction](#mermaid-diagram-plan-construction)
* [Stage 4: Iterative Research - Selecting and Executing Sub-Research](#stage-4-iterative-research---selecting-and-executing-sub-research)
* [4.1: Select Next Items Node](#41-select-next-items-node)
* [4.2: Execute Sub-Research Node (Invoking Server Item Agents)](#42-execute-sub-research-node-invoking-server-item-agents)
* [4.3: Server Item Agent Deep Dive](#43-server-item-agent-deep-dive)
* [4.4: Aggregate Batch Results Node](#44-aggregate-batch-results-node)
* [Mermaid Diagram: Orchestrator's Research Loop](#mermaid-diagram-orchestrators-research-loop)
* [Stage 5: Final Report Synthesis](#stage-5-final-report-synthesis)
5. [Components & Technologies](#components--technologies)
* [Configuration (`app_config.ts`)](#configuration-app_configts)
* [State Management](#state-management)
* [Data Fetching and Caching](#data-fetching-and-caching)
* [Tooling](#tooling)
* [Event System (`events.ts`)](#event-system-eventsts)
6. [Differences from Duo Workflow](#differences-from-duo-workflow)
-----
## 1\. Overview
The **GitLab Research Agent** is an AI-powered system designed to perform in-depth analysis of GitLab artifacts (Epics, Issues, Merge Requests, etc.) and their relationships. It leverages a [Knowledge Graph](https://gitlab.com/groups/gitlab-org/-/epics/17514) (powered by [KùzuDB](https://docs.kuzudb.com/get-started/)) and an agentic framework (LangGraph) to understand user queries, plan research steps, gather information from GitLab, and synthesize comprehensive reports.
The agent architecture is composed of two main types of agents:
* **Orchestrator Agent**: The primary agent that interprets the user's query, plans the overall research strategy, dispatches tasks to Server Item Agents, aggregates their findings, and synthesizes the final global report.
* **Server Item Agent(s)**: Specialized sub-agents responsible for conducting detailed research on individual GitLab items (e.g., a specific Issue or Merge Request). They gather data, analyze it, and produce a focused report for that item.
This document will walk through the lifecycle of a research query, detailing each stage of the process.
## 2\. Core Architecture
### Knowledge Graph Integration
At its core, the agent relies on a [Knowledge Graph](https://gitlab.com/groups/gitlab-org/-/epics/17514) built with [KùzuDB](https://docs.kuzudb.com/). This graph stores GitLab entities (Issues, MRs, Epics, Projects, Groups, Files) as nodes and their interconnections (e.g., `RELATED_TO_ISSUE`, `ISSUE_TO_EPIC`, `MODIFIES_FILE`) as typed relationships.
* **Data Ingestion**: An [indexer component (`knowledge_graph/indexer.ts`)](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/knowledge_graph/indexer.ts) is responsible for populating and updating the knowledge graph. It fetches data from GitLab using the [GitLab API client (`gitlab_api/client.ts`)](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/gitlab_api/client.ts) and a [caching layer (`gitlab_api/fetch_with_cache.ts`)](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/gitlab_api/fetch_with_cache.ts), then transforms and loads it into KùzuDB. The [schema definitions for nodes and relationships are in `knowledge_graph/schemas.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/knowledge_graph/schemas.ts).
* **Data Retrieval**: Agents query the graph using Cypher (via `knowledge_graph/kuzu.ts` ) to discover relevant items and their relationships, providing contextual data for LLM reasoning. Parameterized Cypher templates are used for common query patterns like nearest-neighbor or hierarchy searches.
### Agentic Framework (LangGraph)
The agent's control flow and state management are built using [LangGraph](https://python.langchain.com/docs/langgraph/), a library for building stateful, multi-actor applications with LLMs. LangGraph allows defining the research process as a graph of nodes, where each node represents a specific step or agent.
* [**Orchestrator Agent Graph (`orchestrator_agent/graph.ts`)**](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/orchestrator_agent/graph.ts): Defines the main workflow for handling user queries.
* [**Server Item Agent Graph (`server_item_agent/graph.ts`)**](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/server_item_agent/graph.ts): Defines the workflow for researching individual GitLab items.
State is managed within these graphs, allowing information to be passed between nodes and updated as the research progresses. [Refer to `orchestrator_agent/state.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/orchestrator_agent/state.ts) and [`server_item_agent/state.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/server_item_agent/state.ts) for state definitions.
### Orchestrator and Server Item Agents
As mentioned, the system uses a two-tiered agent design:
* **Orchestrator Agent**: Manages the end-to-end research process. It breaks down the user's query, decides what information is needed, coordinates with Server Item Agents, and compiles the final answer.
* **Server Item Agent**: Acts as a specialist, focusing on a single GitLab item. It uses available tools to gather details, comments, related items (via graph queries), and file content, then synthesizes this information into a report specific to its assigned item.
This separation of concerns allows for parallel processing of item research and a modular approach to adding new capabilities.
## 3\. Overall Execution Flow
The research process, from user query to final report, involves several distinct stages orchestrated by the main agent.
### Mermaid Diagram: Top-Down Architecture
```mermaid
graph TD
D[Orchestrator Agent]
D --> E[Analyze Input & Extract Item IDs]
E --> F{Index Items?}
F -->|Yes| G[Update Knowledge Graph]
F -->|No| H[Create Research Plan]
G --> H
H --> I[Select Items for Research]
I --> J[Execute Parallel Sub-Agents]
J --> K[Aggregate Results]
K --> L{More Research Needed?}
L -->|Yes| I
L -->|No| M[Synthesize Final Report]
M --> N[Stream Response to User]
subgraph "Sub-Agent Research"
J1[Initialize Item Research]
J2[Iterative Data Gathering]
J3[Generate Item Report]
J1 --> J2 --> J3
end
J -.-> J1
```
## 4\. Detailed Execution Stages
The execution of a research query is managed by the [Orchestrator Agent Graph defined in `orchestrator_agent/graph.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/orchestrator_agent/graph.ts).
### Stage 1: Input Analysis & Item ID Extraction
* **Node**: `inputAnalysisNode` ([`orchestrator_agent/nodes/input_analysis.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/orchestrator_agent/nodes/input_analysis.ts))
* **Purpose**: To parse the user's input query, identify any explicitly mentioned GitLab item URLs or references, extract them into a structured format, and determine initial research topics.
* **Process**:
1. The node receives the raw input query from the user (e.g., "What is this POC about? Does it support JetBrains? [https://gitlab.com/groups/gitlab-org/editor-extensions/-/epics/158](https://gitlab.com/groups/gitlab-org/editor-extensions/-/epics/158)").
2. It utilizes an LLM, prompted with a system message to act as a "GitLab Server Item ID Extractor."
3. The LLM is equipped with a tool (`itemIdTool`) whose parameters are defined by `initialInputSchema` (server\_items, reasoning, researchTopics).
4. The LLM processes the query and calls the `itemIdTool` with the extracted item specifications (itemType, iid, fullPath), its reasoning for the extraction, and a list of research topics derived from the query.
5. The extracted item specifications are then converted into `CompleteServerItemId` objects using `buildServerItemIds` from [`item_id.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/item_id.ts).
6. An event `orchestrator_server_item_ids_extracted` is dispatched with the extracted items, reasoning, and research topics.
* **Output**: `InputAnalysisNodeOutput` containing `initialServerItems`, `researchTopics`, and any error. This output is passed to the next node in the orchestrator graph.
#### Reliable GitLab Item ID Extraction
The system uses a combination of LLM understanding and structured tooling to reliably extract GitLab item IDs:
1. **LLM Prompting**: The `inputAnalysisNode` provides a clear system prompt to the LLM, instructing it to identify GitLab items and related research topics from the user's query.
2. **Tool Definition with Schema**: A specific tool, `itemIdTool`, is defined with a Zod schema (`ServerItemToolArgsSchema` from [`item_id.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/item_id.ts) and `initialInputSchema` in [`orchestrator_agent/nodes/input_analysis.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/orchestrator_agent/nodes/input_analysis.ts)). This schema guides the LLM on the expected output format for each item:
* `itemType`: Enum (Issue, MergeRequest, Epic)
* `iid`: String (Internal ID of the item)
* `fullPath`: String (Full path of the project or group, e.g., "gitlab-org/gitlab")
The schema description includes examples of URLs and common textual references for Issues, MRs, and Epics to aid the LLM.
3. **LLM Tool Calling**: The LLM is expected to call this tool with an array of identified items matching the schema.
4. **Structured Conversion**: The arguments provided by the LLM via the tool call are then processed by `buildServerItemIds` ([`item_id.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/item_id.ts)). This function takes the array of `ServerItemToolArgs` and constructs `CompleteServerItemId` objects, which include the globally unique `itemId` (e.g., `gitlab-org/gitlab#123`) generated by `ServerItemIdResolver.buildItemId`.
This multi-step process, combining flexible LLM parsing with strict schema enforcement and standardized ID generation, ensures that various forms of item references from the user query are reliably converted into a consistent, usable format for the rest of the agent's operations.
### Stage 2: Knowledge Graph Indexing (Optional)
* **Node**: `graphIndexNode` ([`orchestrator_agent/nodes/graph_index.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/orchestrator_agent/nodes/graph_index.ts))
* **Purpose**: If the `shouldIndex` flag in the input is true, this node is responsible for indexing the initial items (and their related entities) into the Knowledge Graph. This ensures the graph is up-to-date before proceeding with research.
* **Process**:
1. The node receives the `initialServerItems` from the `inputAnalysisNodeOutput`.
2. It uses the [`Indexer` service (`knowledge_graph/indexer.ts`)](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/knowledge_graph/indexer.ts) configured in [`AppConfig`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/app_config.ts).
3. For each initial item, it calls `indexer.run()`, which fetches the item and its related entities from GitLab and upserts them into KùzuDB.
4. During indexing, `graph_index_progress` events are dispatched to provide feedback on the indexing status.
5. Successfully indexed items are collected.
* **Output**: `GraphIndexNodeOutput` containing `indexedItems` and any error.
* **Routing**:
* If an error occurs, the graph ends.
* Otherwise, it proceeds to the `planNode`.
The [`Indexer`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/knowledge_graph/indexer.ts) itself uses a priority queue (`PQueue`) to manage fetching and processing items. It explores the GitLab data graph starting from the initial items, respecting configured depth limits per item type (`depthByItemConfig`). It fetches data using the [`GitLabDataFetcher`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/knowledge_graph/fetcher.ts), which in turn uses the [`CachedFetcher`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/gitlab_api/fetch_with_cache.ts). Nodes are added to KùzuDB via `kuzu.upsertNode()` and relationships via `kuzu.addRelationshipBulk()`.
#### Mermaid Diagram: Graph Indexing
```mermaid
graph TD
INPUT_ANALYSIS_OUTPUT[From Input Analysis: initialServerItems] --> GRAPH_INDEX_NODE{"Graph Index"};
GRAPH_INDEX_NODE -- For each initial item --> INDEXER_RUN["indexer.run(item)"];
INDEXER_RUN -- Uses --> GITLAB_DATA_FETCHER["GitLabDataFetcher (`fetcher.ts`)"];
GITLAB_DATA_FETCHER -- Uses --> CACHED_FETCHER["CachedFetcher (`fetch_with_cache.ts`)"];
CACHED_FETCHER -- Fetches from --> GITLAB_API[GitLab API];
INDEXER_RUN -- Upserts data to --> KUZU_DB["KùzuDB (`kuzu.ts`)"];
INDEXER_RUN -- Dispatches Events --> EVENT_GRAPH_INDEX_PROGRESS[Event: graph_index_progress];
GRAPH_INDEX_NODE --> OUTPUT_INDEXING[Output: indexedItems];
OUTPUT_INDEXING --> PLAN_NODE[To Plan Node];
```
### Stage 3: Research Plan Construction
* **Node**: `planNode` ([`orchestrator_agent/nodes/plan.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/orchestrator_agent/nodes/plan.ts))
* **Purpose**: To create a high-level research plan. This typically involves exploring the Knowledge Graph starting from the initial items to identify a broader set of relevant items that need to be investigated by the Server Item Agents.
* **Process**:
1. The node initializes a conversation history with a system prompt that instructs the LLM to act as a "GitLab Research Agent." The prompt emphasizes iterative graph exploration using a `graphDB` tool and concluding with a `finalAnswer` tool call that provides the research plan.
2. Initial user message includes details of the `initialServerItems` (fetched using `getFormattedItems` from [`fetch.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/fetch.ts)) and the `researchTopics`.
3. The node enters a loop (up to `MAX_ITERATIONS` ):
* The LLM is called with the current conversation history and available tools:
* `graphDB`: To perform Breadth-First Search (BFS) in the Knowledge Graph. This tool's parameters are defined by `GraphDBToolParamsSchema` (itemIdsToSearch, searchType, searchDepth, reason).
* `finalAnswer`: To signal completion and provide the final research plan text. Its parameters are defined by `FinalAnswerToolParamsSchema` (answer).
* If the `graphDB` tool is called:
* [`fetchGraphDbTool` (from `tools/graph_db_fetch.ts`)](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/tools/graph_db_fetch.ts) is executed with the provided arguments. This function queries KùzuDB (specifically `kuzuService.getItemBFS`).
* The results (or an error) are added to the conversation history as a tool result.
* An `orchestrator_plan_step_generated` event is dispatched.
* If the `finalAnswer` tool is called:
* The provided `answer` (the research plan) is recorded.
* The `isComplete` flag is set to true, and the loop terminates.
* If no tool or an unexpected tool is called, or if `graphDB` is called with empty `itemIdsToSearch`, the system may re-prompt the LLM or conclude the planning phase.
4. The conversation history, loop count, final reasoning (often the research plan itself or an error message), and completion status are saved.
* **Output**: `PlanNodeOutput` containing `reasoning` (the plan), `loopCount`, `conversationHistory`, and `completionStatus` (isComplete, researchPlan).
* **Routing**:
* If planning is not complete or the plan is empty, the graph ends.
* Otherwise, it proceeds to the `selectNextItemsNode`.
#### Mermaid Diagram: Plan Construction
```mermaid
graph TD
PLAN_NODE[Plan Construction]
PLAN_NODE --> ANALYZE_ITEMS[Analyze Initial Items & Research Topics]
ANALYZE_ITEMS --> EXPLORE_GRAPH[Explore Knowledge Graph]
EXPLORE_GRAPH --> DISCOVER_ITEMS[Discover Related Items]
DISCOVER_ITEMS --> DECISION{Found Enough Context?}
DECISION -->|No| EXPLORE_GRAPH
DECISION -->|Yes| CREATE_PLAN[Create Research Plan]
CREATE_PLAN --> PLAN_OUTPUT[Output: Research Plan & Item List]
PLAN_OUTPUT --> SELECT_NEXT_ITEMS_NODE[To Select Next Items Node]
```
### Stage 4: Iterative Research - Selecting and Executing Sub-Research
This is a loop managed by the orchestrator, involving selecting a batch of items for detailed research, dispatching them to Server Item Agents, and then aggregating the results. This loop continues until a conclusion is reached or a maximum number of iterations is hit.
#### 4.1: Select Next Items Node
* **Node**: `selectNextItemsNode` ([`orchestrator_agent/nodes/select_next_items.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/orchestrator_agent/nodes/select_next_items.ts))
* **Purpose**: To decide which GitLab items (from the research plan or previously discovered items) should be researched in the current batch, or if the overall research should conclude.
* **Process**:
1. The node maintains its own conversation history for selecting items, which accumulates over iterations.
2. **Initial Call / Loop Start**:
* If it's the first iteration (empty conversation history), a system prompt is constructed. This prompt instructs the LLM to act as a "research orchestrator," using the `research_plan` from the `planNode` and the original `user_query` to select an initial batch of items. It has two tools: `select_items_for_research` and `end_research`.
* If it's a subsequent iteration, summaries from the `processedItems` of the previous batch (from `aggregateBatchResultsNodeOutput`) are added to the user message, providing new context to the LLM. A continuation prompt asks the LLM to select the *next* batch or end research.
3. **LLM Interaction**:
* The LLM is called with the current selection conversation history and the tools:
* `select_items_for_research`: Parameters defined by `SelectItemsToolParamsSchema` (items\_to\_research, reasoning\_for\_selection, research\_goal\_for\_selected\_items).
* `end_research`: Parameters defined by `EndResearchToolParamsSchema` (reason\_for\_ending\_research, overall\_conclusion).
* The node handles LLM responses, including retries if `InvalidToolArgumentsError` occurs.
4. **Tool Call Processing**:
* If `select_items_for_research` is called:
* The `items_to_research` are converted to `CompleteServerItemId` objects.
* The `research_goal_for_selected_items` is recorded.
* An `orchestrator_items_selected` event is dispatched.
* If `end_research` is called:
* The `overall_conclusion` is recorded, signaling the orchestrator to proceed to final report synthesis.
* If no tool is called or after retries, the system may force a conclusion.
5. The loop count for selection (`state.selectNextItemsNodeOutput.loopCount`) is incremented. If it exceeds `MAX_SELECT_ITEMS_LOOPS`, research is forced to conclude.
* **Output**: `SelectNextItemsNodeOutput` containing `itemsToResearchThisIteration`, `currentBatchResearchGoal`, `reasoning` (which is the `overall_conclusion` if research is ending), and updated `researchSelectionConversationHistory` and `loopCount`.
* **Routing**:
* If `reasoning` (overall\_conclusion) is present, it routes to `synthesizeGlobalReportNode`.
* If `itemsToResearchThisIteration` is not empty, it routes to `executeSubResearchNode`.
* If `MAX_SELECT_ITEMS_LOOPS` is reached or no items are selected and no conclusion is made, the graph ends.
#### 4.2: Execute Sub-Research Node (Invoking Server Item Agents)
* **Node**: `executeSubResearchNode` ([`orchestrator_agent/nodes/execute_sub_research.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/orchestrator_agent/nodes/execute_sub_research.ts))
* **Purpose**: To execute the detailed research for each item selected by `selectNextItemsNode`. This is done by invoking instances of the Server Item Agent graph in parallel.
* **Process**:
1. Receives `itemsToResearchThisIteration` and `currentBatchResearchGoal` from `selectNextItemsNodeOutput`.
2. If no items are selected, it returns an empty `subAgentRunOutputs`.
3. A `PQueue` is used to manage concurrent execution of Server Item Agents (concurrency set to 5).
4. For each `itemToResearch`:
* A `parentAgentConversationHistory` is constructed. This includes:
* System messages with the main user query and the specific research goal for this item.
* Short summaries from previously processed items in the current orchestration cycle (from `state.aggregateBatchResultsNodeOutput.processedItems`) to provide broader context.
* An `ServerItemAgentInput` is prepared, including `itemIdToResearch`, the specific `researchGoal` (or the main query if no specific goal), `parentAgentConversationHistory`, `maxLoops` (default 7 ), and `currentDate`.
* The [`serverItemAgentGraph` (from `server_item_agent/graph.ts`)](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/server_item_agent/graph.ts) is invoked with this input and the application configuration.
* The final state of the Server Item Agent (`finalReport`, `finalReportShortSummary`, `error`) is collected.
5. All results from the sub-agent runs are collected.
* **Output**: `subAgentRunOutputs`, an array where each element contains the `itemIdToResearch`, `finalReport`, `finalReportShortSummary`, and any `error` from a Server Item Agent run. This output is implicitly passed to the `aggregateBatchResultsNode` via the orchestrator state.
#### 4.3: Server Item Agent Deep Dive
Each Server Item Agent, defined in [`server_item_agent/graph.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/server_item_agent/graph.ts), performs a detailed investigation of a single GitLab item.
* **Graph Definition**: `initializeResearchNode` -\> `researchIterationNode` (loops) -\> `finalizeReportNode` (parallel with `generateShortSummaryNode`) -\> `joinServerReportsNode` -\> `END`.
* **1. `initializeResearchNode`** ([`server_item_agent/nodes/initialize_research.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/server_item_agent/nodes/initialize_research.ts)):
* Fetches initial details of the assigned `itemIdToResearch` using `appConfig.cachedFetcher` or `fetchGenericItemDetails`.
* Formats these details into an `initialItemContextString` using `getFormattedItems`.
* Constructs the initial conversation history:
* A system prompt defining its role as a "GitLab Research Sub-Agent," its tools (`graphDB_BFS`, `fetch_comments`, `fetch_mr_file_content`, `fetch_item_details`, `generate_final_report`), and the current date.
* If `parentAgentConversationHistory` is provided by the orchestrator, it's incorporated.
* A user message combining the `researchGoal` and the `initialItemContextString`.
* Initializes state variables like `loopCount`, `isComplete`, `currentResearchData`.
* **2. `researchIterationNode`** ([`server_item_agent/nodes/research_iteration.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/server_item_agent/nodes/research_iteration.ts)):
* This is the main research loop for the sub-agent.
* Increments `loopCount`. If `maxLoops` is reached, it adds a message urging the LLM to call `generate_final_report`.
* Calls an LLM (configured via `getLlmClient` ) with the current `conversationHistory` and a set of tools:
* `TOOL_GRAPH_DB_BFS` ([`GraphDBToolArgsSchema`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/tools/graph_db_fetch.ts)): Executes `fetchGraphDbTool` to query KùzuDB. Results are added to `currentResearchData.bfsResults` and `visitedBfsItemIdsSet` is updated.
* `TOOL_FETCH_COMMENTS` ([`FetchCommentsToolArgsSchema`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/tools/fetch_comments.ts)): Executes `fetchCommentsTool`. Results stored in `currentResearchData.comments`.
* `TOOL_FETCH_MR_FILE_CONTENT` ([`FetchMrFileToolArgsSchema`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/tools/fetch_mr_file_content.ts)): Executes `fetchMRFileTool`. Results stored in `currentResearchData.fileContents`.
* `TOOL_FETCH_ITEM_DETAILS` ([`FetchItemDetailsToolArgsSchema`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/tools/fetch_item_details.ts)): Executes `fetchItemDetailsTool`. Results stored in `currentResearchData.formattedItems`.
* `TOOL_GENERATE_FINAL_REPORT`: Signals readiness to compile the final report. Sets `pendingReportArguments` with the `reasoning_for_completion`.
* LLM response (text and tool calls) and subsequent tool results are added to `conversationHistory`.
* If `appConfig.saveOutput` is true, prompts and responses are saved.
* A `server_item_research_iteration` event is dispatched.
* **Routing**:
* If `pendingReportArguments` is set (by `TOOL_GENERATE_FINAL_REPORT`), it routes to the parallel `finalizeReportNode` and `generateShortSummaryNode`.
* If an error occurs, it ends.
* If `loopCount` exceeds `maxLoops`, it ends.
* Otherwise, it loops back to `researchIterationNode`.
* **3. `finalizeReportNode`** ([`server_item_agent/nodes/finalize_report.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/server_item_agent/nodes/finalize_report.ts)) (runs in parallel with `generateShortSummaryNode`):
* Checks for `pendingReportArguments`. If missing, an error is set.
* Constructs a system prompt for the LLM to synthesize a comprehensive, detailed Markdown report based on the entire conversation history and the provided `reasoning_for_completion`. The prompt asks to include exact copies of relevant code, diagrams, quotes, and comments.
* Calls the LLM to generate `finalReportContent`.
* The report content (or an error message) is added to the `conversationHistory`.
* If `appConfig.saveOutput` is true, the final report is saved to disk.
* Sets `finalReport` in the state and clears `pendingReportArguments` if successful.
* **4. `generateShortSummaryNode`** ([`server_item_agent/nodes/generate_short_summary.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/server_item_agent/nodes/generate_short_summary.ts)) (runs in parallel with `finalizeReportNode`):
* Constructs a system prompt for the LLM to synthesize a *concise* research summary (max 500 words) based on the conversation history and `reasoning_for_completion`. Markdown is not required for this summary.
* Calls the LLM to generate `shortSummaryContent`.
* Sets `finalReportShortSummary` in the state.
* Any LLM error is appended to the main state error.
* **5. `joinServerReportsNode`** ([`server_item_agent/nodes/join_server_reports.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/server_item_agent/nodes/join_server_reports.ts)):
* This node signifies the completion of the sub-agent's work.
* It dispatches a `server_item_research_completed` event containing the `finalReport`, `finalReportSummary`, `conversationHistory`, status, and any error.
* Sets `isComplete` to true in the state.
#### 4.4: Aggregate Batch Results Node
* **Node**: `aggregateBatchResultsNode` ([`orchestrator_agent/nodes/aggregate_batch_results.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/orchestrator_agent/nodes/aggregate_batch_results.ts))
* **Purpose**: To collect the `finalReport`, `finalReportShortSummary`, and `error` status from each Server Item Agent run in the current batch and update the orchestrator's list of `processedItems`.
* **Process**:
1. Iterates through `state.subAgentRunOutputs` (which were produced by `executeSubResearchNode`).
2. For each sub-agent output:
* Determines the status ("success" or "error").
* Creates an `OrchestratorItemResearchSummary` event and dispatches it. This event includes the item ID, type, status, final report, short summary, and error details.
* Appends the result to the `updatedProcessedItems` list in `state.aggregateBatchResultsNodeOutput`.
3. Increments the `loopCount` in `aggregateBatchResultsNodeOutput`.
* **Output**: Updates `aggregateBatchResultsNodeOutput` with the new `processedItems` and `loopCount`. The `currentBatchResearchGoal` and `subAgentRunOutputs` are cleared.
* **Routing**:
* If `selectNextItemsNodeOutput.reasoning` (the overall conclusion) is present, it routes to `synthesizeGlobalReportNode`.
* Otherwise, it loops back to `selectNextItemsNode` for the next iteration.
#### Mermaid Diagram: Orchestrator's Research Loop (Select -\> Execute -\> Aggregate)
```mermaid
graph TD
PLAN_OUTPUT[From Plan Node] --> SELECT_ITEMS_NODE_LOOP_ENTRY{"Select Next Items"};
subgraph Orchestrator Iteration
direction TB
SELECT_ITEMS_NODE_LOOP_ENTRY -- Items & Goal --> EXECUTE_SUB_RESEARCH_NODE{"Execute Sub-Research "};
EXECUTE_SUB_RESEARCH_NODE -- Invokes for each item --> SERVER_ITEM_AGENT[Server Item Agent Graph Instances];
SERVER_ITEM_AGENT -- Returns Reports/Summaries --> AGGREGATE_RESULTS_NODE{"Aggregate Batch Results "};
AGGREGATE_RESULTS_NODE -- Updated processedItems --> SELECT_ITEMS_NODE_LOOP_ENTRY;
end
SELECT_ITEMS_NODE_LOOP_ENTRY -- No More Items / Conclusion Reached --> SYNTHESIZE_GLOBAL_REPORT_NODE[To Synthesize Global Report];
```
### Stage 5: Final Report Synthesis
* **Node**: `synthesizeGlobalReportNode` ([`orchestrator_agent/nodes/synthesize_global_report.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/orchestrator_agent/nodes/synthesize_global_report.ts))
* **Purpose**: To generate a single, comprehensive global report based on the original user query and all the detailed research reports and summaries gathered from the Server Item Agents throughout the process.
* **Process**:
1. An `orchestrator_running_global_report` event is dispatched.
2. It compiles `allDetailedReports` by concatenating the `finalReport` from each successfully processed item in `state.aggregateBatchResultsNodeOutput.processedItems`. If an item failed or had no report, a note is included. If no items were processed, the `planNodeOutput.conversationHistory` is used as context.
3. A detailed system prompt is constructed for the LLM, instructing it to act as a "research report synthesizer." The prompt includes:
* The original `user_query`.
* The `research_plan` and `plan_reasoning` from `planNodeOutput`.
* The `reasoning` from `selectNextItemsNodeOutput` if available (explaining why research might be concluding).
* Formatted initial item details (using `getFormattedItems`).
* The compiled `allDetailedReports`.
* Instructions on formatting (GitLab Flavored Markdown), integrating all data, addressing all aspects of the query, and handling missing information. Specific instructions for formatting links and item IDs are also provided.
4. The LLM is called using `streamText` to generate the `globalReportContent` chunk by chunk.
5. For each chunk received, an `orchestrator_running_global_report_chunk` event is dispatched, allowing for streaming of the final report to the user.
6. If `appConfig.saveOutput` is true, the full prompt and the final report are saved to disk.
7. An `orchestrator_all_research_cycles_completed` event is dispatched with the `finalConclusion` (the global report), `totalItemsProcessed`, and status.
* **Output**: Sets `orchestratorFinalReport` in the state with the generated `globalReportContent` and passes through any errors.
* **Routing**: This is the final node in the main research path, so it proceeds to `END`.
## 5\. Components & Technologies
### Configuration ([`app_config.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/app_config.ts))
* The `AppConfig` interface defines the central configuration for the application.
* It includes services like `KuzuDBService`, `GitLabApiClient`, `CachedFetcher`, and `Indexer`.
* It also manages settings like `saveOutput`, `outputPath`, API keys (`googleApiKey`), preferred LLM provider, and model configurations per node (`modelConfigs`).
* The `ensureAppConfig` function validates and provides default values for the configuration when a LangGraph runnable is invoked. It checks for required services and API keys based on chosen models.
* `getNodeModelConfig` retrieves the appropriate LLM model configuration for a given graph node, falling back to defaults if necessary.
* `getLlmClient` instantiates the correct LLM client (e.g., Gemini or Anthropic) based on the node's model configuration and the `AppConfig`.
### State Management
* **Orchestrator Agent State** ([`orchestrator_agent/state.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/orchestrator_agent/state.ts)): Defines the overall state of the research process, including inputs, outputs of various nodes (input analysis, plan, selection, aggregation), the final report, and error status.
* **Server Item Agent State** ([`server_item_agent/state.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/server_item_agent/state.ts)): Defines the state for an individual item's research, including its input (`itemIdToResearch`, `researchGoal`), conversation history, fetched data (comments, BFS results, file contents, formatted items), loop count, completion status, and reports.
* LangGraph's `StateGraphArgs` and reducers are used to manage how state is updated and merged across different nodes in the graphs.
### Data Fetching and Caching
* **[`GitLabApiClient`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/gitlab_api/client.ts)**: Handles direct communication with the GitLab API.
* **[`CachedFetcher`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/gitlab_api/fetch_with_cache.ts)**: Provides a caching layer on top of the `GitLabApiClient` to reduce redundant API calls and improve performance. It's used by various parts of the system, including the `Indexer` and item detail fetching tools.
* **[`GitLabDataFetcher`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/knowledge_graph/fetcher.ts)**: Used by the `Indexer`, this class is responsible for fetching specific details for different `ItemType`s (Issue, MR, Epic, Project, Group) and transforming them into the schema expected by the Knowledge Graph. It leverages the `CachedFetcher`.
* **[`fetch.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/fetch.ts)**: Contains utility functions like `fetchItemDetails` (generic item fetching) and `getFormattedItems` (formats item data into a string representation for LLM prompts).
### Tooling
The agents use various tools to interact with GitLab data and the Knowledge Graph:
* **Item ID Extraction Tool**: Used by `inputAnalysisNode` (internally, not an explicit tool file).
* **Knowledge Graph Tools**:
* `fetchGraphDbTool` ([`tools/graph_db_fetch.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/tools/graph_db_fetch.ts)): Allows agents (primarily `planNode` and `researchIterationNode`) to perform BFS searches in KùzuDB.
* **GitLab Data Fetching Tools (for Server Item Agent)**:
* `fetchCommentsTool` ([`tools/fetch_comments.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/tools/fetch_comments.ts)): Fetches comments for an item.
* `fetchItemDetailsTool` ([`tools/fetch_item_details.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/tools/fetch_item_details.ts)): Fetches and formats detailed information for specified items.
* `fetchMRFileTool` ([`tools/fetch_mr_file_content.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/tools/fetch_mr_file_content.ts)): Fetches content of files within a Merge Request.
* **Control Flow Tools (for Server Item Agent)**:
* `generate_final_report` tool: Used by `researchIterationNode` to signal completion.
* **Control Flow Tools (for Orchestrator Agent - `selectNextItemsNode`)**:
* `select_items_for_research`: To choose the next batch of items.
* `end_research`: To conclude the entire research.
Each tool typically has a Zod schema defining its arguments (e.g., `FetchCommentsToolArgsSchema`, `GraphDBToolArgsSchema` ).
### Event System ([`events.ts`](https://gitlab.com/michaelangeloio/gitlab-research-agent/-/blob/main/events.ts))
* The system uses a custom event dispatching mechanism (`eventGenerator` using `RunnableLambda` and `dispatchCustomEvent`) to emit various events throughout the research process.
* **Event Types (`AllEvents`)**: A union of all possible event data structures, categorized by Orchestrator events and Server Item Agent events.
* **Orchestrator Events Examples**: `orchestrator_server_item_ids_extracted`, `orchestrator_plan_step_generated`, `orchestrator_items_selected`, `orchestrator_item_research_summary`, `orchestrator_running_global_report_chunk`, `orchestrator_all_research_cycles_completed`, `graph_index_progress`.
* **Server Item Agent Events Examples**: `server_item_research_iteration`, `server_item_research_completed`.
* `FatalGraphError` for critical errors.
* These events are intended for monitoring, logging, and potentially driving UI updates or other external integrations (e.g., streaming updates via the `chat/route.ts` endpoint ).
## Differences from Duo Workflow
The GitLab Research Agent and Duo Workflow, both of which leverage LangGraph, exhibit distinct patterns and serve complementary primary purposes, although they serve different purposes.
### **LangGraph Patterns and Architecture Breakdown**
* GitLab Research Agent:
This agent employs an orchestrator pattern with multiple specialized mini-agents. Its process is geared towards in-depth investigation and knowledge synthesis:
1. **Input Analysis & Planning (`planNode`):** It first analyzes the user's query. A crucial part of its planning involves traversing a **Knowledge Graph** to identify relevant GitLab items (Epics, Issues, MRs, etc.). This stage determines _what_ entities need to be researched.
2. **Creating a Research Plan:** Based on the graph traversal and query analysis, a research plan is formulated, which is essentially a list of specific GitLab items requiring detailed investigation.
3. **Executing Sub-Agent Research:** The orchestrator then dispatches parallel mini-agents. Each of these is a focused mini-agent responsible for conducting iterative, in-depth research on a _single_ GitLab item. These agents utilize a range of tools to retrieve details, comments, code content, and perform additional localized graph queries.
4. **Aggregation and Synthesis:** Finally, the findings from all mini agents are aggregated, and a comprehensive global report is synthesized to answer the initial query.
* GitLab Duo Workflow:
* **Software Development Workflow Pattern:** This involves distinct phases such as context building, goal disambiguation, planning by a `PlanSupervisorAgent`, optional human approval steps (for plans and tools), and execution by a `ToolsExecutor`. The `PlannerAgent` in Duo Workflow focuses on creating a detailed, step-by-step plan of _actions_ for an "engineer agent" to follow to achieve a user's goal (e.g., "create an issue," "edit a file"). The `ToolsExecutor` then executes these pre-defined tasks.
* **Chat Workflow Pattern:** The chat workflow demonstrates a more direct iterative loop: an agent processes input, potentially calls tools via a `ToolsExecutor`, and then the agent processes the results to continue the conversation or call more tools.
### **Architecture Pattern Differences:**
* **Nature of the "Plan":** The Research Agent's plan is a list of _entities to investigate_. Duo Workflow's plan is a sequence of _actions to execute_.
* **Use of Knowledge Graph in Planning:** The Research Agent explicitly uses graph database traversal in its planning phase to discover items. Duo Workflow's planner, based on the provided context, derives its plan from the user's goal and the available tools, rather than relying on graph traversal for planning.
* **Sub-Agent Specialization:** The Research Agent's use of "mini-agents" for in-depth exploration of individual items is a distinct characteristic of its orchestrator pattern, tailored for information gathering and synthesis. Duo Workflow's agents (Planner and Executor) are primarily concerned with defining and then executing a sequence of operations.
### **Research Agent as a Workflow Inside Duo Workflow**
The GitLab Research Agent can be conceptualized as a specialized workflow that could be invoked by a broader system like Duo Workflow.
* **Scenario:** Imagine a Duo Workflow tasked with a complex goal, such as: "Analyze the performance implications of the new caching strategy (Epic-123) and refactor related services if bottlenecks are found."
* **Integration:**
1. Duo Workflow could initiate its process.
2. As part of its "build context" or "planning" phase, it could call the **GitLab Research Agent** as a sub-workflow or a powerful tool. The request would be something like: "Provide a comprehensive report on Epic-123, including all related MRs, issues, discussions on performance, and code modules affected by these MRs."
3. The Research Agent would execute its full process (graph traversal, sub-agent research, synthesis) and return a detailed report.
4. This report then becomes rich, contextual input for Duo Workflow's `PlannerAgent`, enabling it to create a much more informed and effective plan for refactoring. Duo Workflow's `ToolsExecutor` would then carry out the refactoring tasks.
This way, the Research Agent acts as a deep information-gathering service that feeds into the action-oriented capabilities of Duo Workflow.
### **TLDR;**
This highlights the core difference in their intended purpose:
* **GitLab Research Agent:** It is fundamentally designed for **"deep" insights** and **context provisioning**. Its primary output is comprehensive, synthesized knowledge derived from exploring and connecting disparate pieces of information across GitLab. It answers the "what," "why," "how are these connected," and "what is the history." It builds understanding.
* **GitLab Duo Workflow:** It is primarily geared towards **taking action** and **effecting change** within the GitLab environment based on a user's goal. While it provides some context, its primary focus is on executing a plan to achieve a tangible outcome (e.g., creating an issue, modifying a file, or running a CI pipeline).
The Research Agent provides the foundational understanding and rich context that can make other agents, like those within Duo Workflow, **have access to the entire GitLab platform**. Duo Workflow agents can consume the Research Agent's output to make more informed decisions, create more relevant plans, and interact with the user or system with a deeper understanding of the underlying context.
issue