Skip to content

Draft: POC- Implement AI slash command to summarise pipeline jobs failures

issue: #468450

What does this MR do and why?

As part of Verify AI hackathon, a POC to summarise pipeline jobs failures

MR acceptance checklist

This MR implements a /summarize_pipeline_failure slash command to provide a summary of all failed jobs on the current pipeline

Screenshots or screen recordings

Screenshot_2024-06-21_at_17.57.19Screenshot_2024-06-21_at_17.57.15

AI results

For the initial proof of concept, the AI command tool is asked to perform a basic analysis of an array of failed jobs on the current pipeline.

  • the :status, :failure_reason and :job_log used to analyse failed jobs.
  • the ai prompt requests an analysts of job logs for the current pipeline to provide a summary

The following is an example of AI output:

Based on the provided information, the summary of the failed jobs in the current pipeline is as follows:

Summary:

There are 4 failed jobs in the current pipeline. Failures:

Job failure due to a non-existent command:

The job log shows that the command "non_existent_command" does not exist, causing the job to fail with a script failure. Job failure due to a non-zero exit code:

The job log indicates that the job was set to exit with a non-zero status code, which caused the job to fail. Job failure due to a syntax error in the script:

The job log shows that there was a syntax error in the script, specifically in the "if" statement, which caused the job to fail. Job failure due to a missing required file:

The job log shows that the job tried to read a file named "non_existent_file.txt", which does not exist, causing the job to fail. In summary, all 4 jobs in the current pipeline failed due to various script-related issues, such as non-existent commands, non-zero exit codes, syntax errors, and missing required files.

How to set up and validate locally

  1. Setup Duo locally (see the docs from the AI team)
  2. Ensure you have duo active for a local group that has existing pipelines ($ GITLAB_SIMULATE_SAAS=1 RAILS_ENV=development bundle exec rake 'gitlab:duo:setup[<test-group-name>]')
  3. Navigate to a pipeline with failed jobs
  4. Get a clientSubscriptionId - parsing from any duo chat query
  5. Execute a graphQL query
mutation {
  aiAction(
    input: {
      chat: {
        resourceId: "gid://gitlab/Ci::Pipeline/7536",
        content: "/summarize_pipeline_failure",
      }
    	clientSubscriptionId: "85fdac4d-8b53-4bfa-a5ab-f843908c7d2d"
    }
  ){
    requestId
    errors
  }
}
  1. Check duo chat with results on a summary of failed jobs.
Edited by Dmytro Biryukov

Merge request reports