Skip to content

Add Ruby parser to seed the GDK db with the AIEF data

Alexander Chueshev requested to merge ac/add-ai-eval-context-seeder into master

What does this MR do and why?

This MR implements a Ruby parser for the GitLab Data Seeder to provide the GDK instance with relevant data and subsequently evaluate Duo Chat for local changes. Please, find more details in https://gitlab.com/gitlab-org/ai-powered/ai-framework/ai-evaluation-framework/-/issues/5.

Note that we store the dataset outside of this repository at https://gitlab.com/gitlab-org/ai-powered/ai-framework/ai-evaluation-data/-/tree/main/data/duo_chat/v1/jsonl/context/resources?ref_type=heads. Once the database is populated, developers can access the created issues, epics, and other resources in the ai_eval_context group.

Resources:

  • epics
  • groups
  • issues
  • iterations
  • iterations_cadence
  • labels
  • milestones
  • notes
  • projects
  • sourcing_milestones
  • users

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

image

How to set up and validate locally

  1. Clone the dataset repo from https://gitlab.com/gitlab-org/ai-powered/ai-framework/ai-evaluation-data. Please note that the repository has LFS enabled. Refer to the instructions in the README file if you need to clone a specific revision of the dataset.

    AIEF_DATASET_REPO=<set the desired path to store the dataset>
git clone git@gitlab.com:gitlab-org/ai-powered/ai-framework/ai-evaluation-data.git $AIEF_DATASET_REPO

  2. Run the GitLab Data Seeder with the script specified - https://docs.gitlab.com/ee/topics/data_seeder.html.

    # the root user is defined with id=1
AIEF_CONTEXT_DATA=$AIEF_DATASET_REPO/data/duo_chat/v1/jsonl/context/resources bundle exec rake "ee:gitlab:seed:data_seeder[ai_eval_context.rb,1]"

    Each time the developer runs GitLab Data Seeder, it creates a new ai_eval_context group with a unique path.

Closes https://gitlab.com/gitlab-org/ai-powered/ai-framework/ai-evaluation-framework/-/issues/5

Merge request reports