Secure: Explore vocabulary and levels for confidence and severity
Problem:
Today we use a similar vocabulary for communicating Severity and Confidence. This leads to a confusing experience, especially where the two labels appear near to one another. Furthermore, we need to identify what the industry standard for vulnerability scoring is and address any discrepancies we may have in our experience.
Our levels today:
Severity | Confidence |
---|---|
Critical | Confirmed |
High | High |
Medium | Medium |
Low | Low |
Unknown | Unknown |
Info | Ignore |
Undefined | Undefined |
- | Experimental |
Further details
Todo's
-
Understand the industry standard for scoring vulnerabilities -
Research how our vocabulary system for Confidence levels differs or aligns with the industry standard for vulnerability scoring -
Research how our vocabulary system for Severity levels differs or aligns with the industry standard for vulnerability scoring -
Provide insight into how we may achieve alignment between our scoring model and the industry standard.
Findings
More info and details can be found in this comment
- The industry standard for vulnerability scoring is CVSS
- CVSS is in v3 and is being adopted by competitors in the space though there are a few who still align to the CVSS v2 framework.
- Our severity levels do not currently align with the CVSS framework
- Confidence is not a first-class UI element and should not be displayed to the user at this time. details here
- Confidence is a sub-class scoring vector for vulnerability scoring that is used in combination with other sub-class vectors to output a Risk or Severity of a vulnerability.
Proposal
⚠ Note This is not a proposal to use the actual CVSS v3 scoring template and replace our scanner reported data. Simply put, we should adopt the severity framework so we can position our product for the eventuality of the inclusion of features coming along in Category:Vulnerability Management as well as adopt industry best practices.
- Remove confidence from our UI
- Remove Undefined severity level from our reports
- All instances of Undefined severity levels should be translated into
Unknown
.-
Unknown
will become (no severity data) by definition.
-
Confidence level proposal: remove confidence in our UI
Reasoning:
- Confidence is based on our analyzers and we have no control over their logic which isn't well defined in the first place.
- From breakman: "However, Brakeman can easily guess wrong, so it is best to read through all warnings and assess their importance manually." Read a bit more [here. (https://brakemanscanner.org/docs/confidence/)
- Confidence is not a first-class UI entity in relation to assessing vulnerabilities, this is reiterated in CVSS, OWASP and CWSS scoring. Confidence is just one data point that is used when calculating RISK "Severity" of a vulnerability.
- Only ~50% of our analyzers report confidence, which generates screens like the dashboard image above. This is not helpful to our users and has already caused confusion.
- We should want users to verify and assess the vulnerability based on their knowledge of their infrastructure and not trust our reported confidence since our tools do not have knowledge of their system. In these cases where we do report confidence, we risk losing the users' trust by stating a vulnerability has High confidence but it is in a test environment or our analyzer reported a vuln for something that was incorrect.
- Competitors don't show this information for a few reasons:
- They don't want to overprescribe or underprescribed confidence levels since their tools don't have enough information about the user's system.
- If they are relying on CVSS scoring than the reported confidence level (RC) is included as part of the temporal score that aggregates to the Risk score or "Severity".
- Third parties report this data, so if we continue to include confidence in our UI we will have even more "undefined" and "unknown" scores in our product.
- We do not have a definition for confidence in our docs Based on the research, we would not be able to craft an accurate definition because our analyzers either, don't have this information, don't have this information in their docs and/or if they have this information they are using different methods that would make crafting a singular definition all but impossible.
CVSS: quick summary, how confidence factors into vulnerability scoring
CVSS Metric Groups | CVSS Metrics and Equations |
---|---|
![]() |
![]() |
Specifically, the Base equation is derived from two sub equations: the Exploitability sub-score equation, and the Impact sub-score equation. The Exploitability sub-score equation is derived from the Base Exploitability metrics, while the Impact sub-score equation is derived from the Base Impact metrics.
Severity level proposal: combining unknown+undefined into Unknown
Reasoning:
- By aligning with CVSS, we will be following the industry standard for assessing vulnerabilities. OWASP's scoring follows the CVSS scoring approach as well as CWSS.
- Adopting the industry standard will make for an easier translation of data when 3rd parties integrate.
- Allowing for the importing of vulnerability information will be easier since the data will be 1:1 if they are using the industry-standard CVSS framework
- Allowing for the creation of vulnerabilities will become easier since we can provide the user with a CVSS calculator, either in a link or as part of our product. Desired in vulnerability management)
- Using the CVSS framework will reduce the maintenance and change rate of our report formats since we are adopting an industry best practice and not a proprietary scoring system.
- Allowing for the user to change a vulnerability's assessment data will be much easier with the CVSS framework since we can provide them the CVSS calculator to properly score the vulnerability based on their knowledge of their system. Desired in vulnerability management)
- Using an industry-standard system will future proof us. When v4 of CVSS comes out we can address that change much like other competitors have when CVSSv3 came out and they had to manage CVSSv2 scoring as well.
Aligning to CVSS |
---|
![]() |
Why Info
and not None
as CVSS has it?
I am of the opinion that None
as a term in the UI is confusing to users and presents the same problem as undefined
does today. Competitors aligning their scoring to CVSS v3 supplement the None
label in a variety of ways:
- Contrast uses
Info
- Checkmarx uses
Info
- AppScan uses
Info
- OWASP uses
Note
To list a few... So it's best to keep what we have since the only consensus is None
is not the term to use.
Considerations kept in mind for this proposal
- Acknowledge the limitation of our analyzers for this exercise
- Acknowledge the differences between our scanners and analyzer sub-groups
- Past and current efforts:
- Future efforts:
workflowready for development
Goals to accomplish prior to-
UX - Provide information on industry scoring and compare this to our current scoring model -
UX - Provide a recommendation/proposal on how we may address identified discrepancies -
UX - Communicate findings and proposal to devopssecure, ~"devops::defend" and to UX in their respective team meetings. -
Product Management - Conclude if proposal aligns with our near-in and future strategic needs. Ping necessary counterparts to evaluate. -
backend - Evaluate proposal for viability and raise awareness on potential problem areas -
frontend - Evaluate proposal for viability and raise awareness on potential problem areas -
Product Management - Update decisions and outcomes section post evaluation
Decisions & outcomes
1. Remove confidence from our UI
1. Remove the use of `Undefined` in severity level from our reports
1. All instances of `Undefined` severity levels should be translated into `Unknown`.
1. Clearly explain `Unknown` means `no severity data` within the UI and docs.
1. Document how we normalize our severity values into our framework for each analyzer
1. Do the BE work to normalize our values into the new framework
Resources & references:
- https://www.first.org/cvss/specification-document
- https://www.owasp.org/index.php/OWASP_Risk_Rating_Methodology
- https://cwe.mitre.org/cwss/cwss_v1.0.1.html
- Competitive analysis: Severity and confidence scoring for vulnerabilities outdated based on this new information but still holds value