Code Review process for Prompt Engineering

Problem Statement

Prompt development is different than the regular code development and review process, which is well-defined and CI-automated at GitLab.

Current guidelines and examples for Prompt evaluations can be found here:

This issue seeks to distill that information into well-defined steps and potentially automatable steps that can be used by GitLab team members to make Prompt changes.

This issue is required as currently there is no process around what steps must be followed by engineers before a new or updated Prompt can be merged.

Definition of Done

The process is documented in the developer guides.

Scope

This proposal seeks to define the Merge Request approval process for:

  • Creation of Prompts for new AI features
  • Updating of existing Prompts for existing AI features (improvement, bugfixes)
  • Creation of Prompts for existing AI features for new models (groupcustom models only)

Out of Scope

  • This proposal does not cover submission of Prompt changes by GitLab community contributors, which is covered in to be created issue.
  • This issue initially discussed potential efficiency improvements around the evaluation cycle. In order to descope this issue, this conversation has been moved to this to be created issue

Notes

  • It is understood that the Prompt code will move out of Rails and into AI Gateway / Python, but the overall code review process should be similar when that happens.

Additional Context

Proposal

  1. Engineer iterates on a Prompt using their GDK and a local model or via a model deployed in the groups's GCP space
  2. Engineer adds the Prompt to the Rails Monolith with a Merge Request, along with any supporting Ruby changes needed.
  3. Engineer evaluates the Prompt against the Evaluation Framework and adds a comment to the Merge Request, similar to this to be added example
  4. Merge Request is approved and merged by any Backend Maintainer
Edited by Sean Carroll