GitLab issueshttps://gitlab.com/gitlab-org/gitlab/-/issues2023-04-21T19:12:29Zhttps://gitlab.com/gitlab-org/gitlab/-/issues/332879Investigate better way to fetch pipeline for Dependency and License list2023-04-21T19:12:29ZTetiana ChuprynaInvestigate better way to fetch pipeline for Dependency and License list### Problem to solve
For Dependency and License list pages we need [to get the latest pipeline with artifacts of needed type](https://gitlab.com/gitlab-org/gitlab/-/blob/998fe1103b67075ec00c42150287513c1563785e/ee/app/services/security...### Problem to solve
For Dependency and License list pages we need [to get the latest pipeline with artifacts of needed type](https://gitlab.com/gitlab-org/gitlab/-/blob/998fe1103b67075ec00c42150287513c1563785e/ee/app/services/security/report_fetch_service.rb#L14) (:dependency_scanning or :license_scanning). With the recent change of `dependency_scanning` job current way doesn't work correctly (we were fetching just the latest pipeline with needed artifacts, but because `dependency scanning` analyzer creates multiple jobs, some of them are unfinished and not all artifacts are displayed on the page. That resulted in a recent bug https://gitlab.com/gitlab-org/gitlab/-/issues/327955.
During its refinement it was discovered that for Security Dashboard page we get this pipeline [differently](https://gitlab.com/gitlab-org/gitlab/-/blob/b270cb6a0162be20e29eb9a0ae3352de02602cc7/ee/app/models/ee/project.rb#L306). This approach relies on data from the separate table in the database but can't be applied to License List as security artifact types are coupled in this solution.
### Proposal
We need to investigate how we can leverage this approach for Dependency and License list pagesBackloghttps://gitlab.com/gitlab-org/gitlab/-/issues/326011Dependency List redirects to job that doesn't list the dependencies2023-11-20T03:04:16ZFabien Catteaufcatteau@gitlab.comDependency List redirects to job that doesn't list the dependencies<!---
Please read this!
Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label:
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression
- https://gitlab....<!---
Please read this!
Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label:
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=bug
and verify the issue you're about to submit isn't a duplicate.
--->
### Summary
The Dependency List links to a Dependency Scanning job that might not be the one that generates the dependency list.
Beyond that, it links to a single job even when the project pipeline has multiple scanning jobs that generate a dependency list.
This seems to be a regression introduced when replacing the `dependency_scanning` job (using Docker-in-Docker) with multiple Dependency Scanning jobs.
### Further details
The problem is in how the `ReportFetchService` is used to retrieve the `build`, in the `DependenciesController`. This `build` is then passed on top the `DependencyListEntity` to link to the report.
Call trace:
- https://gitlab.com/gitlab-org/gitlab/-/blob/v13.10.0-ee/ee/app/services/security/report_fetch_service.rb#L21
- https://gitlab.com/gitlab-org/gitlab/-/blob/v13.10.0-ee/ee/app/controllers/projects/dependencies_controller.rb#L77
- https://gitlab.com/gitlab-org/gitlab/-/blob/v13.10.0-ee/ee/app/controllers/projects/dependencies_controller.rb#L19
- https://gitlab.com/gitlab-org/gitlab/-/blob/v13.10.0-ee/ee/app/serializers/report_list_entity.rb#L12
- https://gitlab.com/gitlab-org/gitlab/-/blob/v13.10.0-ee/ee/app/serializers/dependency_list_entity.rb#L3
### Steps to reproduce
- Create a JavaScript or Ruby project
- Enable Dependency Scanning (DS) by enabling Auto DevOps or by including the DS CI template
- Go to the Dependency List page
- Click on "Based on the **latest successful** scan"
This redirects to a job that might not be the one that generates a dependency list.
### Example Project
https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium
https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/dependencies redirects to https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/jobs/1106762059, which is a `retire-js-dependency_scanning` job.
### What is the current *bug* behavior?
It redirects to a `retire-js-dependency_scanning` job, even though the report created by the job doesn't list the dependencies; it has no `dependency_files` field.
### What is the expected *correct* behavior?
It should link to jobs that run the Gemnasium analyzers: `gemnasium-dependency_scanning`, `gemnasium-maven-dependency_scanning`, and `gemnasium-python-dependency_scanning`
### Relevant logs and/or screenshots
![Capture_d_écran_2021-03-25_à_17.53.41](/uploads/ffc95e0b989572690b52c722403c7ef8/Capture_d_écran_2021-03-25_à_17.53.41.png)
### Possible fixes
link to the pipeline page instead, as suggested in https://gitlab.com/gitlab-org/gitlab/-/issues/289915
/cc @NicoleSchwartz @plafoucriere @brytanniaBackloghttps://gitlab.com/gitlab-org/gitlab/-/issues/215635Consider a unified component section2023-04-21T19:20:50ZKyle MannConsider a unified component section### Problem to solve
Today, we have a separate dependency list and license compliance sections (referring to project section pages, not MR widget sections); displaying results from two different scanners. However, separating them makes ...### Problem to solve
Today, we have a separate dependency list and license compliance sections (referring to project section pages, not MR widget sections); displaying results from two different scanners. However, separating them makes a segmented experience; whereas they are closely related. The dependency list shows licenses and the license list shows dependencies; understanding how they related together is hard to achieve on separate UI sections. The user is forced to navigate from one list to the other (example: looking into related dependency for out-of-compliance license).
As new features evolve, so will the fragmented UX caused by the information architecture. Consider policies: today there are policies related to license compliance but not dependencies. If/when we add policies related to dependencies, these would be seen in separate areas, given the current info-architecture.
[:film_projector: discussion about upside/downsides](https://www.youtube.com/watch?v=JohsDofXFoo) (:notebook: [related notes](https://docs.google.com/document/d/1v71szVlew4I25LVaKZFhEWf34vnU1tDYRmPlHx2j_hI/edit?usp=sharing))
### Intended users
* [Delaney (Development Team Lead)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#delaney-development-team-lead)
* [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer)
* [Devon (DevOps Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#devon-devops-engineer)
* [Sam (Security Analyst)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sam-security-analyst)
* [Simone (Software Engineer in Test)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#simone-software-engineer-in-test)
### Further details
This supports future goals of 1) showing dependency path/relationship (https://gitlab.com/gitlab-org/gitlab/-/issues/198034 and https://gitlab.com/gitlab-org/gitlab/-/issues/213591), 2) bill of materials for projects (https://gitlab.com/gitlab-org/gitlab/-/issues/214278), 3) compliance gates
Job's to be done:
> As a developer, I want to remove out-of-compliance licenses and understand the corresponding dependency, so I can remove it as needed (to be in compliance)
> When my dependencies have reported vulnerabilities, I want to learn more about the vulnerability cause and implications, so I can make an informed decision on taking action on how to proceed.
> When my organization has a compliance policy with dependencies, I want to be aware if I’m breaking a company policy, so I can make sure my project dependencies are in compliance with my org compliance.
> When I need to audit 3rd party licenses and dependencies, I want to be able to provide inventory of licenses and dependencies for the auditor, so I can have them on record for auditing purposes and be able to share them with auditors and customers.
> When I want to see how dependencies are related, I want to view them grouped by file, so I can access the information faster.
### Proposal
An information architecture revision: a `component` section as an umbrella for 1) dependencies, 2) related licenses, 3) policies
![all](/uploads/0502b908b3a0904fcbb4d35b4c71fce3/all.png)
### Permissions and Security
Same as current
### Documentation
...
### Availability & Testing
...
<!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier.
What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing?
Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance.
* Unit test changes
* Integration test changes
* End-to-end test change
See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning -->
### What does success look like, and how can we measure that?
User:
* Does the user find this section when looking for licenses/dependencies?
* Is the sort by effective is displaying related materials?
* (as policy features evolve) does it help user to apply and view policies in a unified section?
### What is the type of buyer?
~"GitLab Ultimate"
### Is this a cross-stage feature?
Yes, as policies evolve in compliance and defend: related to components may be displayed here: 1) policy awareness, 2) violations
<!-- Communicate if this change will affect multiple Stage Groups or product areas. We recommend always start with the assumption that a feature request will have an impact into another Group. Loop in the most relevant PM and Product Designer from that Group to provide strategic support to help align the Group's broader plan and vision, as well as to avoid UX and technical debt. https://about.gitlab.com/handbook/product/#cross-stage-features -->
### Links / referencesBackloghttps://gitlab.com/gitlab-org/gitlab/-/issues/213685Ignore paths in Dependency List2023-04-21T19:21:17ZPhilippe LafoucrièreIgnore paths in Dependency List<!-- The first three sections: "Problem to solve", "Intended users" and "Proposal", are strongly recommended, while the rest of the sections can be filled out during the problem validation or breakdown phase. However, keep in mind that p...<!-- The first three sections: "Problem to solve", "Intended users" and "Proposal", are strongly recommended, while the rest of the sections can be filled out during the problem validation or breakdown phase. However, keep in mind that providing complete and relevant information early helps our product team validate the problem and start working on a solution. -->
### Problem to solve
<!-- What problem do we solve? Try to define the who/what/why of the opportunity as a user story. For example, "As a (who), I want (what), so I can (why/value)." -->
As a Developer, I want to be able to filter out some paths and ignore the dependencies found in these paths, so my list is not overpopulated with useless data.
### Intended users
* [Delaney (Development Team Lead)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#delaney-development-team-lead)
* [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer)
* [Devon (DevOps Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#devon-devops-engineer)
* [Sidney (Systems Administrator)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sidney-systems-administrator)
* [Sam (Security Analyst)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sam-security-analyst)
### Further details
<!-- Include use cases, benefits, goals, or any other details that will help us understand the problem better. -->
https://gitlab.com/gitlab-org/gitlab/-/dependencies is the list of dependencies for the GitLab project. As of today, this list has 100+ pages of dependencies, some of them in legitimate files (ex: `Gemfile.lock`), and some in test files (`qa/Gemfile.lock`). I should be able to ignore paths like `/qa`, `/specs`, etc.
It gets a bit confusing when considering where this data is coming from: The `dependency_scanning` job is generating the report to fill this page, and there's actually already in place some filtering configured: https://gitlab.com/gitlab-org/gitlab/-/blob/560c71b63cbe85d70a3d1a2d2fa886729b81843b/.gitlab/ci/reports.gitlab-ci.yml#L98
Yet, it also seems to apply to vulnerabilities only.
### Proposal
<!-- How are we going to solve the problem? Try to include the user journey! https://about.gitlab.com/handbook/journeys/#user-journey -->
At least make `DS_EXCLUDED_PATHS` work the same for dependencies.
Maybe exclude from `DS_EXCLUDED_PATHS` by default, but if there's a `DEPENCENDY_LIST_EXCLUDED_PATHS` set, ignore the first one. It might be over-engineered, and a complex UX in the end. To be discussed here.
### Permissions and Security
<!-- What permissions are required to perform the described actions? Are they consistent with the existing permissions as documented for users, groups, and projects as appropriate? Is the proposed behavior consistent between the UI, API, and other access methods (e.g. email replies)?-->
N/A
### Documentation
<!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html
* Add all known Documentation Requirements in this section. See https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements
* If this feature requires changing permissions, update the permissions document. See https://docs.gitlab.com/ee/user/permissions.html -->
Update https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html
### Availability & Testing
<!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier.
What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing?
Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance.
* Unit test changes
* Integration test changes
* End-to-end test change
See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning -->
TBD
### What does success look like, and how can we measure that?
<!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. -->
- Better, more accurate data in the dependency list.
### What is the type of buyer?
<!-- What is the buyer persona for this feature? See https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/buyer-persona/
In which enterprise tier should this feature go? See https://about.gitlab.com/handbook/product/pricing/#four-tiers -->
~"GitLab Ultimate"
### Is this a cross-stage feature?
<!-- Communicate if this change will affect multiple Stage Groups or product areas. We recommend always start with the assumption that a feature request will have an impact into another Group. Loop in the most relevant PM and Product Designer from that Group to provide strategic support to help align the Group's broader plan and vision, as well as to avoid UX and technical debt. https://about.gitlab.com/handbook/product/#cross-stage-features -->
No
### Links / referencesBackloghttps://gitlab.com/gitlab-org/gitlab/-/issues/195934Group npm dependencies by scope2023-04-21T19:22:55ZAnnabel Dunstone GrayGroup npm dependencies by scopeThese are all `babel` dependencies. We should look into grouping these items to avoid clutter. We could also further group items that use the same dependencies across different locations (for example, each component here has a duplicate ...These are all `babel` dependencies. We should look into grouping these items to avoid clutter. We could also further group items that use the same dependencies across different locations (for example, each component here has a duplicate in `qa/`)
![Screen_Shot_2020-01-03_at_1.25.03_PM](/uploads/c6dd8010347eb30a0e1d441f3818187b/Screen_Shot_2020-01-03_at_1.25.03_PM.png)Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/13137Include classification to license name in dependency list2023-02-21T22:41:41ZKyle MannInclude classification to license name in dependency list### Problem
License(s) name will be displayed in the dependency list (#10536). This MVC will only show the license name and anchor to documentation. It does not include the license classification (denied).
### Solution
Include correspon...### Problem
License(s) name will be displayed in the dependency list (#10536). This MVC will only show the license name and anchor to documentation. It does not include the license classification (denied).
### Solution
Include corresponding license classification in the dependency list license data.
### Related Issues
* Ideally completing this issue after/with https://gitlab.com/gitlab-org/gitlab-ee/issues/12937
* https://gitlab.com/gitlab-org/gitlab/issues/12941#note_226297353
### Implementation Plan
#### Backend
* [ ] [Merge software license classifications into the dependency list report](https://gitlab.com/gitlab-org/gitlab/issues/35661)Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/12087Time boxed Engineering Discovery: Dependency List: Show when a component is o...2023-06-27T08:11:20ZAndy VolpeTime boxed Engineering Discovery: Dependency List: Show when a component is out of date#### Background:
In the dependency list, it would be nice to know if I have a component that is outdated so I can update it before it becomes a weakness.
#### Problem:
Out of date components may have weaknesses or vulnerabilities as...#### Background:
In the dependency list, it would be nice to know if I have a component that is outdated so I can update it before it becomes a weakness.
#### Problem:
Out of date components may have weaknesses or vulnerabilities associated with them and vulnerabilities are often reported on or just after the fact when a component update is made available.
#### User:
AS someone tasked with managing the dependency list for my project, I want to know when a component or dependency is out of date so I can update it before it becomes a weakness or contains a vulnerability.
#### Proposal:
- Add a new `out of date` status to the dependency list when components are detected that have not been updated.
- Show which version is recommended to update the component(s) to.Backloghttps://gitlab.com/gitlab-org/gitlab/-/issues/11635List by name/path detected but unsupported dependency files in Dependency List2023-04-21T19:27:09ZMark Florianmflorian@gitlab.comList by name/path detected but unsupported dependency files in Dependency ListIn the Dependency List view, if dependency files are found that are known not to be supported, we display a warning about it:
![Alert-some-or-all-files-not-found](/uploads/7bd86d917ee4efcd0a795f8c2cf57acd/Alert-some-or-all-files-not-fou...In the Dependency List view, if dependency files are found that are known not to be supported, we display a warning about it:
![Alert-some-or-all-files-not-found](/uploads/7bd86d917ee4efcd0a795f8c2cf57acd/Alert-some-or-all-files-not-found.png)
The list shown is of all supported dependency files, but that requires the user to compare their repository against that list themselves to figure out which files are not supported. We can give that information directly to the user by instead listing only those files that have been detected and are not supported.Backlog