subhopam das closed issue #7: Improve Oracle Prompt Templates for Better Answer Quality at GitLab AI Hackathon / Participants / kapillamba4

subhopam das
2026-03-20T01:41:07Z

Overview

The oracle flow uses a static prompt template for answer generation. Improving the prompt with few-shot examples, chain-of-thought reasoning, and structured citation formatting will reduce hallucinations and improve answer quality.

Background

The current prompt in .gitlab/duo/flows/oracle.yaml passes retrieved chunks to the LLM without structured guidance on how to reason over them or cite sources. This leads to answers that may not be grounded in the retrieved context.

Goals

Add few-shot examples demonstrating ideal answer format with citations

Add chain-of-thought (CoT) instructions to guide step-by-step reasoning over retrieved chunks

Enforce structured citation format referencing MR/issue sources (e.g. MR !42, Issue #7)

Add explicit instructions to say "I don't know" when context is insufficient (reduce hallucination)

Experiment with system vs. user prompt separation for better LLM instruction following

Test prompt variants against a golden Q&A dataset

Tasks

Audit current prompt template in oracle.yaml

Design few-shot examples covering MR, issue, and mixed-source queries

Add chain-of-thought reasoning instructions

Add citation formatting instructions aligned with _citation() in query-service/query/main.py

Add "I don't know" fallback instruction when context is insufficient

A/B test new prompt vs. baseline using RAGAS faithfulness and answer relevancy metrics

Document final prompt design and rationale

Acceptance Criteria

Improved RAGAS faithfulness and answer relevancy scores vs. baseline

Answers consistently cite source MRs/issues

Model correctly abstains when retrieved context is insufficient