Idea refinement: automated security release picking
🏚 Problem overview
The development process for including an issue in a security release is to link the security implementation issue to the next security release tracking issue.
This allows the release-tools tooling to pick up the issue. Updates are posted in the security release tracking issue if there are reasons the security implementation issue is not ready or will not be included.
Developers may be developing security MRs at any part of the milestone, but none of them are able to be "ready" until after the stable branch has been created, giving developers about a week of time to create the last remaining backport and fix any other problems preventing their MRs from being "ready".
That week of time aligns with what release managers refer to as the "Early merge phase". During this phase, release managers will run a command multiple times each day to check the readiness of all issues associated with the security release tracking issue. If any are ready, the MR targeting the default branch will be merged. If any are not, they will be listed in a table in the security release tracking issue with some notes about what is preventing them from being ready (MRs are missing, not assigned to the bot, etc...).
It is common practice for release managers to be following the progress of the MRs listed in that table, pinging MR authors and maintainers to see if they need help and are aware of what needs to be done for them to be included in the security release.
Another problem that arrises with the current process is one of the first steps of the security tracking issue workflow is to link the issue to the security release tracking issue. This means that issues are linked before any work has been done. Release managers have to ask the authors if they were intending to be included in the upcoming release, and then usually ask them to unlink and wait for the next issue (or unlink them automatically with our tooling during the normal process).
🛠 Solution overview
🌳 Summary
Change the development process so issues are not linked to the security release tracking issue until they are proven to be ready.
🔍 Details
There are two phases to this:
- Automate a system that finds security issues based on a certain label (in this issue I'll refer to it as
security-target
, but the name can be discussed later), checks their readiness, and then links them to the security release tracking issue when they are ready. - Create a single source of truth (SSOT) for Engineering and Product to use to check the status of their security issues as they are developing them and have decided they want to target an upcoming security release.
⛓ Phase 1 - automated linking
The early merge phase would change. It would look like this:
- Security release tracking issue is opened after a previous release finishes
- Each day a job runs that will:
- Check readiness of all security implementation issues currently linked to the security release tracking issue.
- If the issue is still ready, do nothing
- If the issue is no longer ready, unlink it and post a comment on the security implementation issue, pinging the assignee.
- Find all security implementation issues that have the
security-target
label that are not yet linked to the security release tracking issue. - Check readiness on those issues.
- If the issue is ready, associate it with the security release tracking issue and post a comment on the security implementation issue letting the assignee know it will be included.
- If the issue is not ready, the reason why is communicated in the security implementation issue (possibly pinging the assignee, although, maybe not if this job runs daily)
- When the release manager is ready, they run the normal early merge phase chatops command to merge the default MRs of all linked security implementation issues. Since they are all guaranteed to be ready at this point, the release manager should not have to worry about unlinking any issues and just waits for them to be deployed before moving on.
📓 Phase 2 - SSOT for security issue statuses
- The security release tracking issue will be the SSOT of what is included in the security release, but it would be nice if there was a dashboard where people could see a list of all security implementation issues that details their current status so they do not need to check each one individually.
👍 Benefits
- Release managers only deal with issues that are ready.
- Team members beyond the MR author who want to follow the status of a given issue or all security issues have a place to look for that information.
- Team members are not surprised if their issue is unlinked since it won't be linked until it is ready and they should be aware of that with the process change.
⚠ Risks
- Release managers continue to be pinged to help get MRs ready (hopefully providing good output in the report will prevent this).