Spike: Improvements to SAST Rule Enhancement and Creation Workflow

The Vulnerability Analysis team, and the SAST group, are working towards improving the quality of the security findings our product(s) generate as well as the process responsible for generating them. In order to achieve this, all existing rules are being reviewed and enhanced as well as new rules created or incorporated.

The process involved in thoroughly understanding a vulnerability, creating a Minimal Runnable Example (MRE) which is also vulnerable and writing meaningful rules, generally for our Semgrep analyser, is, to some extent, complex and filled with uncertainty. Some of the questions rule writers and testers need to ask themselves are for example "when does this vulnerabilty arise and when does it not?", "what would a minimal runnable example which is nonetheless realistic look like?" and much more.

This issue is meant to capture some general principles, ideas as well as concrete tasks aimed at improving the efficiency with which rules for our SAST analysers can be created as well as their effectiveness tested and verified.

This issue is a Spike aimed at exploring improvements and opportunities for reducing friction in the rule creation lifecycle.

The following are ideas:

• Formalise and further structure the steps necessary to research, understand, emulate as well as document a vulnerability and rule. This can be achieved, for example, by:

  • Improving upon the SAST-rules Enhancement Checklist under https://gitlab.com/gitlab-org/security-products/sast-rules/-/blob/main/docs/enhance-rule-checklist.md. Initial issue and MR templates draft aimed at achieving this can be found at https://docs.google.com/document/d/14pK8sCE2Z4jR1rJ9Uv5pvenISFqP-VP0rtcKMjbCE-g/edit
  • Maintaining SAST rule creation and improvement guidelines which help contributions remain consistent ( gitlab-org/security-products/sast-rules!354 (diffs) )

• Make vulnerability proof of concepts and their associated code the source of truth during local rule writing and rule testing as well as in down-stream CI/CD workflows. This has mainly two objectives. First, it would improve the ease with which vulnerable scenarios are replicated, ran and extended. Secondly, runnable vulnerable proof of concepts are partially self-documenting, more so when comment annotations are used in the relevant positions and these can be reused for testing and coverage verification without preparing or distilling separate, disjoint and potentially out-of-sync test cases. Initial first steps could be, as described in #439434 (closed), to establish a test stage in the CI/CD pipeline for proof of concept applications in https://gitlab.com/gitlab-org/security-products/tests

• Added criteria for the evaluation of internal and externally sourced MREs !143268 (diffs)

• Make better use of GitLab, mainly issues and merge request as well as their associated labels, to track the state before, during and after rule requests/creation/porting/validation. An initial step towards a more concrete rule lifecycle and workflow can be to create a dashboard in order to track development and validation work similar to https://gitlab.com/groups/gitlab-org/-/boards/1420731?label_name[]=group%3A%3Asecurity%20policies .

• Consistent use of labels as per https://handbook.gitlab.com/handbook/engineering/workflow/ reduces overhead related to state tracking and allows sharing state and progress asynchronously among the team members and external collaborators. An initial PoC for a SAST Rule Improvement Workflow Board can be found under https://gitlab.com/groups/gitlab-org/-/boards/7309853?label_name[]=Category%3ASAST&group_by=epic&label_name[]=group%3A%3Avulnerability%20research . We should likely focus on the labels workflowproblem validation , workflowready for development , workflowin dev , workflowin review , Category:SAST , groupvulnerability research , ~"sast-rule-enhancement" , featureaddition , featureenhancement , devopssecure , typefeature

  • Review rule enhancement issues created in the past and re-label them in order to accurately represent progress so far, especially those issues labeled groupstatic analysis

• Added SAST Ruleset Improvement Board and label usage to the SAST Ruleset Enhancement issue template in !143268 (diffs)

• Determine, document and discuss the true positive, true negative, false positive and false negative rates of each rule before and after creation or enhancements have been conducted. The aforementioned data about any given rule is crucial in order to inform the development process of the vulnerable proof of concept and its associated rule(s). This data, both granular and aggregated, can be used for further investigations, as the basis improvements, etc.

• Set-up a reviewer roulette process similar to that used by gitlab-org/gitlab in order to share the load among the team and reduce uncertainty and friction while communicating with external contributors.

cc @gitlab-org/secure/vulnerability-research @wayne as per our conversations last week