Skip to content

[Chain of Custody Report] Expand the scope from Merge Commits to All Commits

Michael Becker requested to merge feat/267601 into master

What does this MR do and why?

Issue 267601 requests to expand the chain of custody report from Merge Commits to All Commits.

The meaning of "all" is a little bit vague, so I will expand on the implementation below.

First thing to mention is that this change is feature flagged with the all_commits_compliance_report flag. This will allow us to iterate on the implementation with input and feedback from production use.

Background

The compliance report prior to this change is a CSV report that contains info from all Merge Requests under a given group, in the past month.

The report is generated in a worker, and emailed to the requesting user. There is a 15mb max limit on the CSV due to the fact it is being emailed.

Implementation

Since there is a 15mb limit, it doesn't really make sense to grab all commits, especially for a group with a lot of active projects, as they will ultimately be truncated.

The current strategy I took is:

  1. Get all projects under a group
  2. Iterate over all projects, grabbing all repository commits 2a. I am only grabbing from a 1-month window 2b. I am only grabbing COMMITS_PER_PROJECT max commits
  3. combine all of those commits into a collection
  4. feed them into the CSV builder, letting it truncate at 15mb wherever that falls at

The existing "merge commit filter" feature

There is a feature to generate this report for a specific MR sha. That feature is still "MR Only", but will also be expanded to all commits once the new version of the report has baked in production a bit

Getting a merge request from a gitly commit

This feature is supposed to be "expanding" the existing report, not replacing it. As such, it seems like we should keep the existing columns and populate them whenever a gitly commit can be mapped to an MR

The only way I really found to do that is using the existing MR finder while iterating over every commit. This seems "not very efficient" to me, but I couldn't find another way to map from gitly commits to MRs

Supporting changes

CommitCollection

In order to be able to pass a CommitCollection to the CsvBuilder library, there needs to be a find_each method defined. Since repository commits are not actually active record object, here we just define a find_each method that aliases the CommitCollection#each method

Screenshots & Example Reports

before after
33-merge-commits-1668464947.csv 33-commits-1668464879.csv
Screenshot_from_2022-11-14_16-07-18 Screenshot_from_2022-11-14_16-06-37

How to set up and validate locally

  1. Enable FF 
    Feature.enable(:all_commits_compliance_report)
  2. You must have an EE enabled instance
  3. go to a group you are the owner of
  4. select "security & compliance -> compliance report" from the left panel nav
  5. click the "list of all commits" button
    • note: you will need some recent (within a month) commits
  6. after report is generated, you can see the email in http://localhost:3000/rails/letter_opener/
  7. to actually inspect the CSV file, navigate to the gitlab/tmp/letter_opener/ directory

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Michael Becker

Merge request reports