FY23-Q1 SBOM Survey Insights
Study
Understand summary usage information and preferred terminology regarding Dependencies, Software Packages, and SBoMs.
View the research request issue for additional details.
Design
A Qualtrics survey was created and distributed through social media, with a prompt to target an audience that would fit the target profile. The same survey was also distributed in the GitLab application, accessible to users who visited either a security dashboard or the dependency list.
Target profile:
how we defined cohorts in the study
The survey automatically screened based on one attribute:
Does your role require you to manage or interact with software packages/components, third party libraries, dependencies and/or their resulting inventory list?
Participants who answered yes
, or I am not sure, but I am familiar with those terms
were allowed to proceed. Participants who screened into and completed the survey are defined as Valid completes
Target cohorts
During the survey design, we identified two distinct cohorts who match our known personas for this problem area;
- Engineers/Developer: Individual contributors and Team leads/managers
- Security: Individual contributors and Team leads/managers. (due to low census in this group, all security participants n=13 have been included in the Security cohort.
- Top-line: This is not a cohort, rather this represents all participants n=69 and will be referred to during the summary.
Cohort | Profile | # | % of participants |
---|---|---|---|
Top-line | All participants | 79 | - |
Engineering | All roles | 54 | 68.3% |
Security | All roles | 13 | 16.5% |
Other | All roles | 12 | 15.2% |
Participant makeup
- 79 valid completes of 207 responses (38% completion rate)
- ~90% of responses were from GitLab users
- Time to field: 86 days
See all participant data
Role | Org Position | # | % |
---|---|---|---|
Individual contributor | Engineering / Development | 27 | 34.2% |
Manager/Team lead | Engineering / Development | 21 | 26.5% |
Director | Engineering / Development | 2 | 2.5% |
C-Level/Sr. Leader | Engineering / Development | 3 | 3.7% |
Board Member | Engineering / Development | 1 | 1.2% |
Individual contributor | Security | 1 | 1.2% |
Manager/Team lead | Security | 9 | 11.4% |
Director | Security | 2 | 2.5% |
C-Level/Sr. Leader | Security | 1 | 1.2% |
Individual contributor | Infrastructure | 4 | 5% |
Manager/team lead | Infrastructure | 2 | 2.5% |
Individual contributor | Customer success | 1 | 1.2% |
Manager/team lead | Customer success | 1 | 1.2% |
Manager/team lead | Product | 1 | 1.2% |
Individual contributor | Quality | 1 | 1.2% |
Manager/team lead | Quality | 1 | 1.2% |
C-Level/Sr. Leader | Quality | 1 | 1.2% |
Summary
Key Themes
- Terms in this topic (dependencies, packages, libraries, SBoM, etc) are variable and subject to the application, underlying software, and even the user's background.
- The SBoM needs to be flexible and configurable to meet the needs of companies with different security maturities/needs, compliance requirements, DevOps processes, and release cadences.
High-level takeaways
Feature naming:
- Participants were not in agreement on what to call this feature, though
Dependency list
(41%) had the highest response count overSBoM (30%)
.
Direct and indirect dependency naming
- The preferred terms for external software packages directly included were
Dependency (38%)
followed byThird-party [dependency/package] 21.5%
. - The preferred terms for external software packages indirectly included were
Transitive Dependency (22.8%)
andDependency (20%)
, the latter being with no qualifier, just Dependency on its own.
Software vs. Container
- A majority of participants
71%
refer to both application software dependencies and container dependencies using the same term. - A majority of participants who refer to these with the same term prefer them
Dependencies (80%)
, which when looking at all responses to the question, represents 50% of all participant's answers.
Search values
- The most common values participants mentioned:
Package name (59 74.9%)
,Version (32 40.5%)
,License (15 20%)
,Vulnerabilities (10 12.6%)
andRelated/Nested Dependencies 8 (10.1%)
SBoM Generation & Lifespan
- With no clear consensus on how, where, and when an SBoM should be generated - top two responses being
On every build 35 (30.7%)
andOn default/main branch builds 30 (26.3%)
- we should consider the need for flexibility when designating the SBoM generation. - A majority of participants
41 (51%)
responded that their org/team defines how long their SBoM should exist, which alludes to the need for user-defined lifespans for SBoMs. - A relative majority of participants expect their SBoM to exist for
1 year +, 35 (44.3%)
with the remaining participants somewhat split between the other options ranging from 6 months to 2 weeks or less.
Results
Q1: What term do you use for external software packages you include in your project (usually through a package manager)?
Answer:
There is a slight preference for Dependency List
at a high level. However, a pattern has emerged based on the application of the tool, where {IC Engineers} have a preference for Dependency List
and {Security} + {Engineering Managers} have a preference for SBoM
. A weak assumption can be made that the (user's task + their role) plays a factor in their naming choice.
Note: we did not have any {Security + IC} participants in the valid, complete pool, therefore we cannot assume with a high degree of confidence the significance of (user's task + their role) plays in the naming preference.
Supporting data
Data:
Most answers were split between Dependency list (41%)
and SBoM (30%)
. Another notable preference was for Installed packages (18%). Reviewing the data shows a visible pattern between cohorts and their preferences.
Cohort Result matrix (# of responses):
Cohort | SBoM | Dependency list | Installed packages | Component inventory | Other (open text) |
---|---|---|---|---|---|
Engineers | 10 | 25 | 12 | 4 | 3 |
Security | 9 | 3 | 1 | 0 | 0 |
Other | 5 | 4 | 1 | 1 | 1 |
Role result matrix (# of responses) For Engineering + Security Cohorts
Role | SBoM | Dependency list | Installed packages | Component inventory | Other |
---|---|---|---|---|---|
Individual contributor | 4 | 20 | 7 | 1 | 2 |
Manager/Team lead | 17 | 8 | 6 | 2 | 1 |
*Sr Leader + | 3 | 4 | 1 | 1 | 1 |
*Sr. Leader + (C-Level/Sr. Leader, Directory, Executive, Board Member)
Q2: What term do you use for external software packages you include in your project (usually through a package manager)? (Open text response)
Answer:
There is a weak preference for Dependency (38%)
followed by Third-party [dependency/package] 21.5%
across the responses. No significant patterns emerged when comparing cohorts, roles, or departments for this question.
Supporting data
Data:
75% of responses fall into these groups
- Dependency - 30
- Third party " " - 17
- External " " - 9
- Direct " " - 3
25% of responses where no pattern could be inferred
Other terms mentioned: component, package, package manager, modules, lib/package, artifact.
Q3: What term do you use for software packages indirectly linked to your project that are included by an external package of code you directly added?
Answer:
From the responses, two answers stood out; Transitive Dependency (22.8%)
and Dependency (20%)
, the latter being with no qualifier, just Dependency on its own. Looking closer at the results, we see the mention of transient and tertiary, which can be lumped into a group (transient/tertiary/transitive/third party 30.3%
). This would be the most prominent response group; however, across all the answers, we do not see strong agreement for this term. No significant patterns emerged when comparing cohorts, roles, or departments for this question.
Supporting data
Data:
57% of the responses fall into these groups:
- Transitive/transient/tertiary/thrid party: 24 (30.3%)
- Transitive " "- 18
- Thrid party " " - 4
- Transient " "- 1
- Tertiary " " - 1
- Dependencies - 16 (20.3%)
- Indirect " " - 5 (6.3%)
43% of the responses where no pattern could be inferred
Other notable Dependency "qualifiers": [recursive, external, child, sub,]
Other notable terms which don't use "dependency": [external software, packages, library]
Q4: Do you use different terms to refer to application software dependencies vs. container dependencies?
Answer:
A majority of participants 71%
refer to both application software dependencies and container dependencies using the same term. Of the responses who refer to both with the same term, 39 (80%)
of the group and 50%
of total responses refer to these both as Dependencies
.
Supporting data
Data:
Cohort | Different terms | Same terms |
---|---|---|
All | 23 (29%) | 56 (71%) |
Engineering | 13 (16.5%) | 41 (52%) |
Security | 4 (5%) | 9 (11.4%) |
Other | 6 (7.6%) | 6 (7.6%) |
Other notable terms for (Same terms):
- Application dependencies
- Product dependencies
Notable terms for (different terms):
- Software Dependencies v. Container Dependencies - 6
- Application Dependencies v. Container Dependencies - 9
Q5: Typically, what values do you use to search your inventory of software components and dependencies? (ex, Package name, license, package version, etc.,.) (free text response)
Answer:
The most common values participants mentioned: Package name (59 74.9%)
, Version (32 40.5%)
, License (15 20%)
, Vulnerabilities (10 12.6%)
and Related/Nested Dependencies 8 (10.1%)
Supporting data
Data:
- Package Name - 59
- Version - 32
- License - 15
- Has vulnerabilities (Vulns/ issues / security / CVEs etc.,) - 10
- Nested dependencies (required by, related, other " " etc,.) - 8
Other notable values; URL, Path, Tags, Coordinates, Activity, Status, Up-to-date.
Q6: When would you expect the inventory of software components and dependencies to be generated and included as part of the build package?
Answer:
There is no clear consensus among the participants as it relates to when the SBoM is generated. The top two responses On every build 35 (30.7%)
and On default/main branch builds 30 (26.3%)
, which accounts for 57% of all responses. With no definitive preference, we should consider the need for flexibility when defining SBoM generation.
Supporting data
Data:
Cohort | Every build | On default branch | Branches w/specific tags | Protected branches | Product Branches | Other |
---|---|---|---|---|---|---|
All | 35 (30.7%) | 30 (26.3% ) | 19 (16.7%) | 16 (14%) | 9 (7.9%) | 5 (4.4%) |
Engineering | 22 | 23 | 15 | 13 | 5 | 3 |
Security | 9 | 3 | 2 | 1 | 0 | 0 |
Other | 4 | 4 | 3 | 3 | 2 | 2 |
Q6a: (Follow up): How do you typically designate product branches? please explain.
Answer:
Only 9 participants met the criteria to answer this question by answering Product Branches
in the previous question, 1 of which did not appropriately respond to the question. Given the low census n=8, the information provided is not directional at this time.
Supporting data
Data:
Open response text:
For deliverable software artifacts: tagged with specific release versions. For cloud service: "prod".
We name the branch 'Release-[major.minor.patch?.hotpatch]' as needed. It would also work to use labels to designate branches as 'product' branches, that's another good option
directory style branch name (production/product/version)
Environment based and then release version branches
I designate them according to each client. EX: IBM-PT
cli
{username}-{version}
"prod", meaning the currently publicly-visible build
Q7: Typically, how long do you expect your inventory of software components and dependencies to exist after it has been generated?
Answer:
A relative majority of participants responded with 1 year +, 35 (44.3%)
with the remaining options closely distributed around the 12% to 18% for each. An assumption can be made that the SBoM lifespan requirements relate to the compliance needs of organizations as well as their release cadence.
Supporting data
Data:
Cohort | 1 year + | 6 months | 3 months | 1 month | 2 weeks or less |
---|---|---|---|---|---|
All | 35 (44.3%) | 10 (12.66%) | 9 (11.39%) | 10 (12.66%) | 15 (18.9%) |
Engineering | 22 | 8 | 7 | 7 | 10 |
Security | 6 | 1 | 2 | 1 | 3 |
Other | 7 | 1 | 0 | 2 | 2 |
Q8: What defines how long your inventory of software components and dependencies exists?
Answer:
A majority of participants responded the team/organization defines this period 41 (51%)
indicating the need for a flexible SBoM lifespan that is user defined.
Supporting data
Data:
Cohort | User defined | System defined | Uncertain |
---|---|---|---|
All | 41 (51.9%) | 19 (24.1% ) | 19 (24.1%) |
Engineering | 25 | 16 | 13 |
Security | 8 | 2 | 3 |
Other | 8 | 3 | 1 |
- User defined = The team/organization defines this period
- System defined = The application/system defines the period
Additional thoughts (captured at the end of the survey)
Expand to see additional thoughts
a bill of materials should exist as long as the build result exists. in best, the BOM is bundled into the build result, so both co-exist or die at the same time.
Currently (as far as I know), your package and vulnerability scans do not allow for the organization or project to temporarily ignore any package vulnerabilities. This would be a very nice feature for when there are no current "solution" or fixed versions available for a specific external package. Currently the only options are to permanently ignore this vulnerability, or leave it and constantly have an F rating. However it would be nice to have a regular period of 30 days or so to ignore that vulnerability for a specific reason (version lock, no fix available, etc.), but to have it regularly check back to make sure it is still something the team/project wants to ignore.
CycloneDX support across all builds with the ability to have different retention based on configurable factors such as branch naming pattern, protected branches, labels, etc
https://dependencytrack.org/ https://cyclonedx.org/use-cases/
I wish there was a more "central" list for both dependencies and for the security issues that some dependencies might have
Ideally need something out-of-band of pipelines, that can be toggled on. Vulnerabilities and Security Dashboard needs to become more performant for large namespaces!
Nowadays keeping containers up-to-date and knowing where those are deployed is the challenge.
SBOM is a good means to understand what is there in the application developed . But have alone a dependency SBOM my not serve the the complete purpose of IP and Security risk. Any SCA tool which gives a possibility to look in the source code and see different license issues and identifying them as a component will always be a added value to SBOM.
SCA must be used strategically also, at the security architecture/design stage.
The list view would benefit from some search/filtering options instead of just sorting
Triggering updates from an overview list (and seeing possible updates on the first place) would be nice
Try to adopt an international standard (or multiple standards) for the SBoM, like CycloneDX
We currently use CycloneDX as our SBOM format. It would be great to have native support in GitLab for it.
We have standardized on Package URL. It is a requirement for any container or SCA scanning solution. We have also standardized on CycloneDX. So any solution we purchase (or continue to renew) must have these two.
We need SBoM s to be compliant to standards such as cybersecurity and UNECE
What should be included in SBOM? Does it include third party vendor software, agents you run etc.? How do you combine SBOMs? Do you need to combine SBOMs of your vendors with yours? How deep in the supply chain is reasonable?
Related to #299 (closed).