Creation of a RCA prompt library
Problem to Solve
As we start looking into evaluation of Root Cause Analysis (RCA) , we will be first working on creating a customised dataset as proxy to production to be able to evaluate:
- Foundational Models
- RCA Features
- Quality Benchmark when ground truth is preset and when it is not
Implementation Plan
We will be looking into our extractor tool to extract data from
Evaluation Data:
- Job API and validate the fields needed from it( DRI @HongtaoYang, @mfanGitLab )
- Source Code ( DRI @srayner ) . This is already work in progress as for Explain This Vulnerability (ETV)
Feature Pipeline
- Simultaneously we will look into Graph QL API to extract analyse_ci_failure ( @tle_gitlab ) . This is similar to our pipeline for slash commands as established
Additional Details
- Extractor Source: https://gitlab.com/gitlab-org/modelops/ai-model-validation-and-research/ai-evaluation/vulnerabilityexplanation/-/blob/main/fetch_async.py?ref_type=heads
Progress
June 8: We plan to extend the dataset from 409 prompts to further delve into areas where it performs low and different classifications of pipeline errors next week as well as looking into scheduled runs.
Edited by Susie Bitters