Auto Remediate dependency security vulnerabilities
Problem
It's great to detect security vulnerabilities in dependencies, and provide instructions for how to update the dependency to a secure version, but if the dependency has already been patched, why make users do any manual work? Instead of emailing about all the unsecure dependencies, how about just emailing that they've all been updated automatically?
Use Case
Debian has a security fix for openSSH. Organizations without GitLab walk into the office and see that 1000 applications need to be updated. Organizations with GitLab walk into the office and see that 999 applications were updated successfully and 1 had to be rolled back. We can do this because we have monitoring integrated into GitLab, we're not deploying into a black hole, but can see if the application still meets the service level objectives.
Here's a complete flow:
- Start with a project with a Kubernetes cluster.
- Use web terminal and local editor to create something.
- Push to trigger Auto DevOps (with auto deploy to production, automatic (parallel) tests, and review apps).
- Debian package used in your container has a security fix.
- GitLab auto detects you have that dependency and creates a new build.
- The changelog is updated automatically.
- In the merge request you can see the metrics and SLO (service level objective) was not met and after deploying to 1% it was reverted.
- All this happened automatically after Debian pushed to stable.
- Luckily when you check the next day your application is not vulnerable.
- So you use chat to close the issue that was automatically opened after the failed deploy.
- An early user of your service also emailed GitLab service desk to ask if the Debian security update is addressed.
- You create a blog post via the web terminal that is published with GitLab pages.
Proposal
- Periodically review dependencies for security vulnerabilities or used push-based notifications of changes in common dependencies to quickly detect affected projects.
- Prepare code changes to mitigate affected projects as merge request (MR).
- Set MR to merge-when-pipeline-succeeds.
- If pipeline succeeds, MR will automatically be merged and deployment will start (if project is set up with CD).
- If deployment succeeds, notify that the vulnerability has been remediated.
- If SLO alert is triggered during incremental rollout (or any other failure is detected), rollback the deployment and auto-revert the MR (so
master
is in a known good state. (Known auto-reverting should also trigger a redundant deploy of essentially the same code, but the first rollback is faster so still beneficial).- Recreate MR and send notification that we tried to patch, but a problem was detected in production, rollback and revert were done, provide information on vulnerability, etc.
- If pipeline fails, notification is sent that we tried to patch the project, with information about the vulnerabilities, link to MR, etc.
Links
- [meta] VersionEye integration
- New site: shield.gitlab.com
- GitLab License finder
- VersionEye - Display in Merge requests list view
- VersionEye - Display information in the Merge request details view
- VersionEye - Display in Commits list view
- VersionEye - Add option to activate GitLab Version Service to the administration panel of a repository
- VersionEye - Settings panel for GitLab Version Service
- Prior art: https://depfu.io/
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.