Skip to content

Use direct answer instead of resource reader

Mark Chao requested to merge 450915-ai_prompt_current_page_skip_reader into master

What does this MR do and why?

When current resource is available (epic/issue), we force LLM to use it directly instead of using IssueReader/EpicReader. This would save one LLM call, making current page related queries faster.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Evaluation results - Independent LLM Judge - Correctness

  • Before: Latest result from daily production evaluation (master)
  • After: This MR (450915-ai_prompt_current_page_skip_reader - SHA: 9ce1e6df96f7fb0dab9bfcf3cc88b7e6ebf48002)
after_percentage before_percentage grade
61.3 54.8 4
17.2 15.1 3
1.1 3.2 2
7.5 14.0 1
query
WITH grades as (
  SELECT 4 as grade union all
  SELECT 3 as grade union all
  SELECT 2 as grade union all
  SELECT 1 as grade
), before_base_table AS (
  SELECT *
  FROM `dev-ai-research-0e2f8974.duo_chat_external_results.lulalala-master-0328_20240328_002952__independent_llm_judge`
  WHERE answering_model = 'duo-chat-local'
), after_base_table AS (
  SELECT *
  FROM `dev-ai-research-0e2f8974.duo_chat_external_results.lulalala-master-0327_20240327_230534__independent_llm_judge`
  WHERE answering_model = 'duo-chat-local'
), before_correctness_grade AS (
  SELECT correctness as grade, COUNT(*) as count
  FROM before_base_table
  GROUP BY correctness
), after_correctness_grade AS (
  SELECT correctness as grade, COUNT(*) as count
  FROM after_base_table
  GROUP BY correctness
)

SELECT grades.grade AS grade,
       ROUND((COALESCE(before_correctness_grade.count, 0) / (SELECT COUNT(*) FROM before_base_table)) * 100.0, 1) AS before_percentage,
       ROUND((COALESCE(after_correctness_grade.count, 0) / (SELECT COUNT(*) FROM after_base_table)) * 100.0, 1) AS after_percentage,
FROM grades
LEFT OUTER JOIN before_correctness_grade ON before_correctness_grade.grade = grades.grade
LEFT OUTER JOIN after_correctness_grade ON after_correctness_grade.grade = grades.grade;

Evaluation results - Similarity score

after_percentage before_percentage similarity_score_range
6.5 2.2 1.0
64.5 57.0 0.9
20.4 24.7 0.8
3.2 4.3 0.7
3.2 7.5 0.6
2.2 4.3 0.5
0.0 0.0 0.4
0.0 0.0 0.3
0.0 0.0 0.2
0.0 0.0 0.1
query
WITH buckets as (
  SELECT 1.0 as bucket union all
  SELECT 0.9 as bucket union all
  SELECT 0.8 as bucket union all
  SELECT 0.7 as bucket union all
  SELECT 0.6 as bucket union all
  SELECT 0.5 as bucket union all
  SELECT 0.4 as bucket union all
  SELECT 0.3 as bucket union all
  SELECT 0.2 as bucket union all
  SELECT 0.1 as bucket
), before_similarity_score AS (
  SELECT *
  FROM `dev-ai-research-0e2f8974.duo_chat_external_results.lulalala-master-0328_20240328_002952__similarity_score`
  WHERE answering_model = 'duo-chat-local'
), after_similarity_score AS (
  SELECT *
  FROM `dev-ai-research-0e2f8974.duo_chat_external_results.lulalala-master-0327_20240327_230533__similarity_score`
  WHERE answering_model = 'duo-chat-local'
)

SELECT buckets.bucket AS similarity_score_range,
    (
        SELECT ROUND((COUNT(*) / (SELECT COUNT(*) FROM before_similarity_score)) * 100.0, 1)
        FROM before_similarity_score
        WHERE buckets.bucket = ROUND(before_similarity_score.comparison_similarity, 1)
    ) AS before_percentage,
    (
        SELECT ROUND((COUNT(*) / (SELECT COUNT(*) FROM after_similarity_score)) * 100.0, 1)
        FROM after_similarity_score
        WHERE buckets.bucket = ROUND(after_similarity_score.comparison_similarity, 1)
    ) AS after_percentage,
FROM buckets

Screenshots or screen recordings

N/A

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Related to #450915 (closed)

Edited by Mark Chao

Merge request reports