Deprecate build support on Dependency Scanning and CI based security scanning with Gemnasium
Summary
In Discuss new UX for enabling Dependency Scanning... (gitlab#458920 - closed) we discussed the various approaches we might take to improve the user experience for enabling Dependency Scanning once the SBOM based Dependency Scanning capability will be achieved. The key outcomes of the discussion were:
- Deprecate and remove Dependency Scanning builds
- Create CI/CD Components for Dependency Scanning
Another important part of this plan is the removal of CI based security scanning with the Gemnasium analyzer and the focus on SBOM based security scanning as part of our default offering.
Deprecation of build support
To summarize, the strategy going forward is:
- Use a Dependency Scanning analyzer to parse lockfiles or equivalent to generate an SBOM artifact
- Rely on native SBOM generators to create an SBOM artifact to be provided to us
- Rely on generic SBOM generators to create an SBOM artifact to be provided to us
This epic depends on delivery of the first item, which is tracked in Software composition analyzer for dependency gr... (&14484 - closed). The others will be pursued separately and likely after this is complete.
Adoption of CI/CD Components
The existing use of CI templates relies on rules that detect the project type in order to run the appropriate jobs.
This epic depends on the CI/CD Component that enables our new analyzer, which supports monorepos and doesn't require template rules to select the appropriate job because there's only one job with the default configuration. This work is tracked in Create Dependency Scanning CI/CD Component (gitlab#433267 - closed).
The CI/CD components have not deemed to be mature enough yet to support this transition. See Update Security template guidance to prefer com... (gitlab#489904)
Deprecation of CI based security scanning with Gemnasium analyzer
The new DS analyzer will only we responsible for generating an SBOM report artifact and no security scan will be done within the CI job. Instead, the Dependency Scanning feature will leverage the scanning capabilities developed for Continuous Vulnerability Scanning, which are executed within the rails platform.
This changes comes with important side-effects and feature removals that must be carefully reviewed by users. We will list them exhaustively as part of the deprecation announcement.
WIP list:
- auto-remediation feature for yarn will not be available with the new analyzer and thus removed as of 18.0 unless we managed to replace it with a new implementation (&759) before that release.
- DS scans of vendored JS libraries will not be available with the new analyzer and thus removed as of 18.0 unless we managed to replace it with a new implementatin (&7186) before that release.
- DS security report will no longer be generated by the built in DS feature as of 18.0, impacting any existing workflow based on these CI artifact files, either to modify them before they get ingested in GitLab or to consume them in another CI job or an external vulnerability management system. The ability to download results via the UI is also lost. The suggested replacement approach is to use the GraphQL API. See Ensure the GraphQL API provides an acceptable U... (gitlab#512514 - closed) for ongoing validation.
Scope
This epic covers the necesary work to deprecate the existing features, and prepare our users for the transition to the new Dependency scanning by using SBOM feature.
This includes:
- Identify and announce the deprecated features
- Define the migration plan from Gemnasium to SBOM generator (i.e. the "new DS analyzer")
- Identify gaps and decide if they need to be addressed before the deprecation announcement (in %17.9 at most), or the removal (in %18.0) tracked in Transition to Dependency Scanning with SBOM for... (&15727)
Migration plan
At a high-level, the current proposal to migrate users from Gemnasium to the SBOM generator involves leveraging the DS CI Templates (stable and latest).
- Understand if and how to support MR pipelines with the CI/CD component.
- The
latest
CI template is the de-facto support for MR pipelines for DS (related: MR pipeline support in Sec templates (gitlab#410880)). While we are allowed to make breaking-changes to this template, we should an effort to provide an alternative.
- The
- Devise a way to guide users through the process of generating lock file(s) for their project.
- The SBOM generator will not write a report for projects that require a build. We don't want the DS jobs to fail silently.
- Ensure the CI templates changes work with scan execution policies and pipeline execution policies.
- There's no need to include compliance pipelines as they were deprecated in %17.3 and are being removed in %18.0.
- Modify the
latest
template to include the CI/CD component by %17.9 (but sooner is better). - Overwrite the
stable
template withlatest
on %18.0. - Consider whether we want to deprecate the CI template and remove them in 19.0 since, at this stage, it's mostly a wrapper around the CI/CD Component.
Activity
added issue gitlab#467039 (closed)
mentioned in issue gitlab#467039 (closed)
mentioned in merge request gitlab-org/security-products/analyzers/gemnasium!790 (merged)
marked this epic as related to &7288 (closed)
@johncrowley, I've linked Full dependency graph support in new component ... (&7288 - closed) as it implements the functionality we need to deprecate the Gemnasium equivalent.
I haven't set it as a blocker because we may choose to deprecate support for each language in Gemnasium as the new version becomes available.
While we're on the topic, the epic above will make Allow all Java and Python files to be scanned (&12313 - closed) obsolete.
/cc @hacks4oats
mentioned in issue gitlab#467445 (closed)
mentioned in epic &14125 (closed)
mentioned in issue gitlab#471766 (closed)
marked this epic as related to &14145
removed the relation with &14145
mentioned in epic &14627 (closed)
added epic &14627 (closed) as parent epic
added issue gitlab#474906 from epic &14627 (closed)
mentioned in issue gitlab#475275 (closed)
removed parent epic #14627 (closed)
added #14627 (closed) as child epic
added #14636 (closed) as child epic
removed child epic #14636 (closed)
marked this epic as blocked by #14636 (closed)
mentioned in issue gitlab#411410 (closed)
mentioned in issue gitlab#428876 (closed)
added workflowplanning breakdown label
removed the relation with &7288 (closed)
marked this epic as blocked by &7288 (closed)
mentioned in issue gitlab#482903 (closed)
assigned to @fcatteau
assigned to @gonzoyumo and unassigned @fcatteau
TODO: create issue. It seems CVS on SBOM changes is not calling the
MarkAsResolvedService
. This means CVS on sbom change does not update existing findings from previous execution that are no longer detected, which is necessary before we can stop generating DS Security Report in the CI based scans.EDIT: Mark vulnerabilities as no longer detected on t... (gitlab#498726 - closed)
Edited by Olivier Gonzalezmentioned in issue gitlab#498726 (closed)
added gitlab#498726 (closed) as child issue
Drop or keep CI templates?
@johncrowley the migration plan will depend on the outcome of Update Security template guidance to prefer com... (gitlab#489904). Specifically, do we recognize the impact of replacing CI templates with CI/CD components for customers (see gitlab#489904 (comment 2115705213)) and is the Sec Product team willing to follow this direction?
I can look for both options (we keep the CI templates or we drop the CI templates) but that will be more work and time spent while we have to make the above decision anyway. Could you please help moving this forward?
Edited by Olivier Gonzalez@gonzoyumo, the discussion you linked to is 3 weeks old and several screens long.
I'm concerned that, at this pace, we'll miss the due date for delivery of this epic. It is very important that we remove build support or we can't remove Gemnasium, which would mean keeping it for another year.
The maintenance burden of Gemnasium is significant, and will be compounded by upcoming challenges that I can't disclose publicly. This would leave the CA group in a tight spot.
Could you please summarize what you are asking of John? Even better if you can make a recommendation.
@thiagocsf sure. The TLDR is: do we keep CI templates as the default enablement solution for our Security features, or do we switch to CI/CD components instead?
The currently suggested migration plan relies on the assumption that we're keeping the CI templates, but that seems to diverge from the general intent at GitLab and what's proposed in Update Security template guidance to prefer com... (gitlab#489904)
From this FAQ blog post: https://about.gitlab.com/blog/2024/08/01/faq-gitlab-ci-cd-catalog/
Will existing CI/CD templates be migrated to components?
Yes, the GitLab templates are migrated and have a special badge in the CI/CD Catalog.
What's the recommended way to transition from our existing GitLab pipeline templates to GitLab catalog components?
This should be rather simple since components are very similar to templates. We would recommend start using inputs in your templates, and later on moving them to the appropriate folder structure.
I'd like the Product team in sectionsec to confirm we want align the section on that direction (stop using the CI templates) by
18.0
(May 2025) or take a step back for now and keep them until at least19.0
(May 2026).The concerns raised haven't been answered so far therefore my recommendation is to keep the CI templates until we've confirmed the CI/CD components on their own are a suitable and well accepted entrypoint for customers of security features (ultimate mostly).
@gonzoyumo, thank you!
I'm supportive of keeping the templates because they're the simplest way to migrate to the the new analyzer.
If we can avoid repetition by adding the component in the template, even better.
Finally, we might want to add some logic to the template to provide guidance to users with projects that would formerly trigger a build.
It makes sense to stagger this roll-out:
- 18.0: use CI templates as a transition to new DS experience. Ideally rely on Components.
- 19.0: follow grouppipeline authoring advice for removal of CI templates (if removing, we'll need to deprecate by 18.9 but hopefully sooner).
I think we keep the templates to ease the migration for users. I don't want to introduce too many changes at once for our users. We can eventually migrate to components.
Related to this: gitlab#458920 (comment 1934703377)
mentioned in epic #14146
mentioned in issue gitlab#410880
Metrics
@johncrowley we should review which product metrics we are currently using that are based on the existing Dependency Scanning CI jobs (e.g. xMAUs?). Particularly, if anything is based on the dependency scanning report artifact, or any related downstream record like the
[Security::Scan](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/security/scan.rb)
model, then this will likely no longer work when switching over SBOM base scans only.As part of SBOM-based dependency scanning findings for def... (&8026 - closed) we've added the specific usage metric
cvs_on_sbom_change
but I'm not sure this is enough. Could you please advise on the product requirements for usage metrics?Edited by Olivier GonzalezAfter a quick glance, I've found a couple of instrumentation classes that seem to be used for secure metrics and are based on criteria that will no longer apply to SBOM based scans:
Instrumentation class Metrics CountCiBuildsMetric
counts.dependency_scanning_jobs
CountSecurePipelinesMetric
usage_activity_by_stage_monthly.secure.dependency_scanning_pipeline
CountSecurityScansMetric
usage_activity_by_stage_monthly.secure.dependency_scanning_scans
usage_activity_by_stage.secure.dependency_scanning_scans
I'd expect at least these two Tableau dashboards to be impacted:
Usefull Metrics Dictionary with a group filer: https://metrics.gitlab.com/?status=active&productGroup=composition_analysis
Note that the
count of scan type
already has a substitute for dependency scanning in the additional charts asCVS on SBOM changes
.For
Number of Findings per Report by Scan Type
there is no direct equivalent in SBOM scanning. I suppose we could start tracking the number of components listed in the SBOM report.I wouldn't consider these essential for the progress of this epic, but it's @johncrowley's call.
@thiagocsf thanks. These susbtitutes are good for our own group to monitor the feature usage and health but I'm concerned about metrics like xMAUs which roll up to upper levels, and up to the company level MAUs I assume.
From internal handbook, it seems
user_unique_users_all_secure_scanners
metric is what drives SMAU for Secure stage, which uses theCountUsersCreatingCiBuildsMetric
instrumentation class which is based on the CI job name If that's the case, then it's already brokenI agree this is not strictly a blocker for progressing with the deprecation itself but I think we must be prepared for when we stop generating DS security reports from our default configuration which is why I'm flagging it here.
For
Number of Findings per Report by Scan Type
there is no direct equivalent in SBOM scanning. I suppose we could start tracking the number of components listed in the SBOM report.Couldn't we just count how many vulnerabilities (or findings when &14636 (closed) is done) are created from a given SBOM scan? Tracking components could be another data point indeed but this might maybe fall within Category:Dependency Management instead?
Couldn't we just count how many vulnerabilities (or findings when &14636 (closed) is done) are created from a given SBOM scan?
Isn't this what we have in
CVS on SBOM changes
? i.e. theKNOWN_AFFECTED_SBOM_OCCURRENCES
dimension.@thiagocsf yeah it's somewhat giving the same information but I wonder how much we would prefer to keep such metrics in the same place to compare with other security features?
Ultimately we could still distinguish what's coming from CVS vs the CI based scanner thanks to the
scanner
property.NB: this will also depend on the discussion in &14636 (comment 2164717688).
mentioned in issue gitlab#489904
mentioned in issue gitlab#412353
mentioned in issue gitlab#474906
mentioned in epic #14636 (closed)
mentioned in issue gitlab#499525
2024-10-23 Status update
I'm diving deep into all the related epics and issues, collecting decisions and ensuring we're all set for the deprecation, so we can write an exhaustive announcement and define a migration plan. There are a couple of pending decisions that can influence the user impact. BTW I'd like to emphasize that the epic is currently focused on the removal of build support on DS but user impact goes beyond that:
- auto-remediation feature for yarn will not be available with the new analyzer and thus removed as of 18.0 unless we managed to replace it with a new implementation (&759) before that release.
- DS scans of vendored JS libraries will not be available with the new analyzer and thus removed as of 18.0. Pending decision: &11617 (comment 2174249052))
- DS security report will no longer be generated by the built in DS feature as of 18.0, meaning any existing workflow based on these CI artifact files, either to modify them before they get ingested in GitLab or to consume them in another CI job or an external vulnerability management system. Pending partial solution: &14636 (comment 2026706118)
@johncrowley I wonder if it would make sense to take a more generic route and call out the deprecation of the overall Dependency Scanning feature as it exists today
. This would probably allow to better highlight the change of paradigm (we no longer do security scanning in the CI) and cover all the impacts that result of that change. This might also be particularly usefull for customers who are currently onboarding and setting up the existing DS feature. They might want to already look toward the new one depending on their need rather than investing on something that won't stay. cc @billydowningI'll continue to investigate and create the necessary issues to try locking the scope ASAP.
2024-10-31 Status update
- I've merged the child epic Deprecate and remove Gemnasium (&14627 - closed) into this one do simplify the organization. As mentionned above, all that work is tightly coupled and must happen at the same time.
- We've decided to try doing a single announcement: gitlab#501308
- We've nearly completed the analysis of user facing impacts and how we're envisioning to address them.
-
Several issues have been added highlighting gaps in CVS on SBOM changes versus the existing workflow based on the Dependency Scanning Security Reports. These will likely have to be addressed to provide a complete User Experience matching our standard. There might still be some leeway into how we prioritize them, though. I might need some help to close the newly identified issues (rails). - Next week I will dive into the CI template update which will then allow to write the migration tutorial.
- A separate epic has been created for the actual removal: Transition to Dependency Scanning with SBOM for... (&15727). Identified work that is not strictly necessary for the deprecation is rather moved there, either as a child issue or as a blocker item.
Edited by Olivier Gonzalez2024-11-07 Status update
I haven't done much progress on the CI template update as I've been sick and got sidetracked to other tasks.
- a recent support request highlighted a new user impact of the transition to SBOM based scanning. The new issue is being evaluated: gitlab#502469 (closed)
- the fact that we use the same scanner name for multiple types of scans (DS, CS, CS for registry) might cause additional problems with the vulnerability management system. An identified occurrence is being discussed in gitlab#498726 (comment 2194819399) but other part of the product must be checked.
2024-11-14 Status update
- Mark vulnerabilities as no longer detected on t... (gitlab#498726 - closed) is now refined and ready for dev
-
Update Dependency-Scanning.latest.gitlab-ci.yml... (gitlab#501103 - closed) is in refinement but I'm already facinga challenge. It sounds like we can't include a CI/CD within a CI template
I'll continue investigating and come up with a POC to validate the migration proposal.
2024-11-21 Status update
- Proposal for the template update: gitlab#501103 (comment 2220673119)
Edited by Olivier Gonzalez2024-12-13 Status update
- Update Dependency-Scanning.latest.gitlab-ci.yml CI template to replace Gemnasium jobs with the new DS analyzer PoC validated and refinement is completed. The work will be executed and release in 17.9, when we annouce the deprecation.
- Docs: Write Dependency Scanning migration tutorials for all supported projects in progress. A draft MR is open to validate the direction and the suggested format: gitlab!175029 (merged)
Edited by Olivier Gonzalez
mentioned in issue gitlab#500716
mentioned in issue gitlab#501087 (closed)
added gitlab#501087 (closed) as child issue
mentioned in issue gitlab#501092
added gitlab#501092 as child issue
mentioned in epic #14627 (closed)
added gitlab#501103 (closed) as child issue
added gitlab#500551 (closed) as child issue
added deprecation label
mentioned in epic #15727
mentioned in issue gitlab#501308
added gitlab#501308 as child issue
mentioned in issue gitlab#476224 (closed)
marked this epic as blocked by gitlab#476224 (closed)
removed the relation with #7288 (closed)
added gitlab#501475 (closed) as child issue
mentioned in issue gitlab#501443
marked this epic as related to #15875
mentioned in issue gitlab#482904 (closed)
mentioned in epic #15960 (closed)
marked this epic as blocked by #15960 (closed)
marked this epic as blocking #15727
Document how to access results from security scans
As we'll no longer provide security scans results in a CI job artifact, we need to clearly document what is the best approach to retrieve these results in a programatic way.
This is usefull for users who want to extract this information into their own Vulnerability Management system or for reporting purposes for instance.
@rdickenson I believe this kind of information will be relevant to keep in the main Dependency Scanning documentation, and not only in a migration guide. E.g. having a dedicated section or page like "How to acces DS security scan results?" WDYT?
@gonzoyumo I agree that we should retain this in the main dependency scanning docs page.
As part of a docs epic I'm trying to split off the "Security scan results" documentation into its own page. For details, see Docs overhaul - Move docs of security scan resu... (gitlab!173946 - merged). Though that page should link to other security scanning tools' documentation as required.
For the title, I think it would be good for the reader if we used the same title across all the security scanning tools' docs pages - "Access security scan results". WDYT?
Thanks @rdickenson. This MR looks great! "Access security scan results" is fine by me.
mentioned in issue gitlab#508411 (closed)
added gitlab#508784 as child issue
added gitlab#509783 (closed) as child issue
added gitlab#509826 (closed) as child issue
added gitlab#510198 as child issue
mentioned in merge request gitlab!176101 (merged)
mentioned in issue gitlab#490348 (closed)
mentioned in issue gitlab#482906
mentioned in issue gitlab#432396 (closed)
Hello @thiagocsf!
I'm seeking clarification on this. When Gemnasium is deprecated and removed in a few months, is the new pattern to use Dependency scanning by using SBOM, Continuous Vulnerability Scanning, both? If DS by SBOM, I'm assuming we expect the feature to be GA by GitLab 18.0? A customer is looking to establish a forward-looking practice right now and they want to get ahead of Gemnasium's removal.
Thank you!
Hi @brad.
We indeed intend on replacing the Gemnasium based Dependency Scanning feaure with the SBOM based one. We are currently writing a migration guide to explain the transition: gitlab!175029 (diffs)
Continuous Vulnerability Scanning is complementary to either of the above feature and will not be affected by the transition. We just will reuse the same vulnerability scanning engine between SBOM based DS and CVS.
We also expect to reach GA before the removal. The GA epic &15961 is blocking the removal epic: &15727
mentioned in issue gitlab#439540 (closed)
mentioned in merge request gitlab!178084 (merged)
added gitlab#514587 as child issue
mentioned in epic #16311
mentioned in merge request gitlab!178678 (merged)
added gitlab#517653 (closed) as child issue
added gitlab#519597 (closed) as child issue
mentioned in issue gitlab-com/support/support-team-meta#6659
added gitlab#515324 as child issue
mentioned in issue gitlab#515324
added typefeature label
added backend label
removed issue gitlab#510198
mentioned in issue gitlab#519328
A feedback/question based on this internal request: Does the SBOM-scanner support approval policies related to newly detected vulnerabilities in an MR? The customer is currently using querying
Pipeline.securityReportFindings
via GraphQL within a job. But they noticed that the resource is updated only after the pipeline completes. This would prevent custom jobs that explicitly fail when new vulns are discovered, correct? If so, the necessary fall-back would be blocking the MR via a policy or approval rule.@katrinleinweber-gtlb exactly.
The built-in approach with GitLab has never been to fail a pipeline to prevent proceeding when vulnerabilities are found. The MR approval policies are what we offer to gate changes based on security scans results.
That said, since that workflow was possible, some customers used it. To prevent disrupting such workflow for customers who can't migrate shortly, we have decided to continue generating the JSON report file from the existing Gemnasium job even past 18.0. These results won't be uploaded to the GitLab platform and won't affect the vulnerability management system but they can still be used to maintain that kind of custom workflow if cusomers need more time to migrate.
With Dependency Scanning using SBOM this will no longer be possible as indeed the security analysis is done in the platform and after the CI pipeline is complete.
mentioned in epic #13469 (closed)