[Chain of Custody Report] Expand the scope from Merge Commits to All Commits
What does this MR do and why?
Issue 267601 requests to expand the chain of custody report from Merge Commits to All Commits.
The meaning of "all" is a little bit vague, so I will expand on the implementation below.
First thing to mention is that this change is feature flagged with the
all_commits_compliance_report
flag. This will allow us to iterate on
the implementation with input and feedback from production use.
Background
The compliance report prior to this change is a CSV report that contains info from all Merge Requests under a given group, in the past month.
The report is generated in a worker, and emailed to the requesting user. There is a 15mb max limit on the CSV due to the fact it is being emailed.
Implementation
Since there is a 15mb limit, it doesn't really make sense to grab all commits, especially for a group with a lot of active projects, as they will ultimately be truncated.
The current strategy I took is:
- Get all projects under a group
- Iterate over all projects, grabbing all repository commits
2a. I am only grabbing from a 1-month window
2b. I am only grabbing
COMMITS_PER_PROJECT
max commits - combine all of those commits into a collection
- feed them into the CSV builder, letting it truncate at 15mb wherever that falls at
The existing "merge commit filter" feature
There is a feature to generate this report for a specific MR sha. That feature is still "MR Only", but will also be expanded to all commits once the new version of the report has baked in production a bit
Getting a merge request from a gitly commit
This feature is supposed to be "expanding" the existing report, not
replacing it. As such, it seems like we should keep the existing
columns and populate them whenever a gitly
commit can be mapped to
an MR
The only way I really found to do that is using the existing MR finder while iterating over every commit. This seems "not very efficient" to me, but I couldn't find another way to map from gitly commits to MRs
Supporting changes
CommitCollection
In order to be able to pass a CommitCollection
to the
CsvBuilder
library, there needs to be a find_each
method
defined. Since repository commits are not actually active record
object, here we just define a find_each
method that aliases the
CommitCollection#each
method
Screenshots & Example Reports
before | after |
---|---|
33-merge-commits-1668464947.csv | 33-commits-1668464879.csv |
![]() |
![]() |
How to set up and validate locally
- Enable FF
Feature.enable(:all_commits_compliance_report)
- You must have an EE enabled instance
- go to a group you are the owner of
- select "security & compliance -> compliance report" from the left panel nav
- click the "list of all commits" button
- note: you will need some recent (within a month) commits
- after report is generated, you can see the email in http://localhost:3000/rails/letter_opener/
- to actually inspect the CSV file, navigate to the
gitlab/tmp/letter_opener/
directory
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.