Creation of a RCA prompt library

Problem to Solve

As we start looking into evaluation of Root Cause Analysis (RCA) , we will be first working on creating a customised dataset as proxy to production to be able to evaluate:

1. Foundational Models
2. RCA Features
3. Quality Benchmark when ground truth is preset and when it is not

Implementation Plan

We will be looking into our extractor tool to extract data from

Evaluation Data:

1. Job API and validate the fields needed from it( DRI @HongtaoYang, @mfanGitLab )
2. Source Code ( DRI @srayner ) . This is already work in progress as for Explain This Vulnerability (ETV)

Feature Pipeline

1. Simultaneously we will look into Graph QL API to extract analyse_ci_failure ( @tle_gitlab ) . This is similar to our pipeline for slash commands as established

Additional Details

1. Extractor Source: https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/vulnerabilityexplanation/-/blob/main/fetch_async.py?ref_type=heads

Progress

June 8: We plan to extend the dataset from 409 prompts to further delve into areas where it performs low and different classifications of pipeline errors next week as well as looking into scheduled runs.