GitLab report artifacts can contain redundant information and excess spaces (due to indentation) that have a negative impact in terms of space and parsing efficiency. In this issue we explore possible space and performance improvements we can potentially achieve by applying various optimizations to the JSON artifacts. The list below shows modifications we can apply to the baseline (indented) JSON artifacts that were emitted by various analyzers in order to remove redundant information. The following list explains some optimizations we can apply to our JSON artifacts: 1. unindent: generate a flat version of the JSON artifacts without any white-spaces 1. remove empty `remediations` array 1. remove redundant end line from vulnerability location: if `start_line` and `end_line` are identical, remove `end_line` 1. ~~remove redundant links: some analyzers produce a large number of redundant `urls` and attach them to our analyzers. For example container scanning attaches many links to a finding, that are available through the actual CVE advisory and are therefore redundant. With this optimization, we only add links that are relevant: source-code repository, CVEs.~~ - created a [separte issue](https://gitlab.com/gitlab-org/gitlab/-/issues/339882) for this because this is an analyzer-specific change. 1. ~~remove redundant vulnerability `name`: some analyzers set exactly the same values for `name` and `message` of a vulnerability; since the `name` does not seem to serve a particular purpose, we can delete it~~ - left this out for the time being because this could unintended side-effects depending on how `name` is used. 1. ~~remove `cve` field for vulnerabilities: the `cve` field is deprecated so we can remove it from the reports produced by our analyzers altogether~~ - this optimization cannot be applied at the moment because it involves backend changes https://gitlab.com/gitlab-org/gitlab/-/issues/209850 # Evaluation The goal of this evaluation is to get an idea of lower and upper bounds of the performance and space improvements we can expect when omitting indentation. The details of the evaluation (setup, test-data, scripts) can be found in the [evaluation issue](https://gitlab.com/gitlab-org/secure/vulnerability-research/research/experiments/-/issues/9). We applied the optimizations mentioned in the list above on different report artifacts (DS, SAST, Container Scanning) using the `oj` parsing gem. We measured three properties: 1. file size (uncompressed): the reduction of the uncompressed JSON file-size in comparison to the baseline 1. file size (compressed): the reduction of the compressed (ZIP) JSON file-size in comparison to the zipped baseline 1. parsing speed: the reduction of (file reading + loading) parsing time in comparison to the baseline Below you can find the summarized results from the evaluation, averaged over multiple runs on 10 baseline, indented JSON artifacts that were emitted by various analyzers. The detailed results are available [here](https://gitlab.com/gitlab-org/secure/vulnerability-research/research/experiments/-/issues/9#note_655981878). Column `unindent` shows the improvements for optimization 1 whereas column `all` shows the results when applying all optimizations mentioned above. The improvements are expressed in terms of ranges (`X%-Y%`, where `X` denotes the min expected improvement and `Y` denotes the max expected improvement in comparison to the baseline). Note that the level of improvement depends on the input file and the level of redundancy produced by the different analyzers. | | unindent (optimization 1) | all (optimization 1-6) | | ------ | ------ | ------ | | file size (uncompressed) | 20%-40% | 34%-53% | | file size (compressed) | 5%-12% | 5%-41% | | parsing speed | 1%-8% | 3%-41% | Stripping spaces/unindenting the file in isolation improves the space efficiency by 20% to 40%, for compressed files the improvement ranges between 5% and 12%. The parsing speed is improvement ranges from 1% to 8%. If we apply all the optimizations mentioned above, the space efficiency improvement ranges from 34% to 53%; for compressed files we can get improvements between 4% and 41%. The parsing speed improvement ranges from 3% to 41%. The objective of this issue is to see how much performance we can gain by applying very simple changes to our JSON artifact serialization (changes that can be applied within a couple of days) to reduce parsing-time and space requirements shown in the table above. # Proposal This is an issue we probably have to tackle on the analyzer's end. There are two different strategies how we could tackle this issue: 1. Change the serialization for every analyzer to implement the optimizations mentioned above. We could include this change directly in the common libraries that are shared across multiple analyzers. 2. ~~Develop a post-analyzer that applies the optimizations.~~ The first solution may be more straightforward for analyzers that are maintained by us. However, the second solution has the advantage that we could use it for custom-analyzers as well. <!-- start-discoto-summary --> ## Auto-Summary :robot: <details> <summary>Discoto Usage</summary> --- > **Points** > > Discussion points are declared by headings, list items, and single > lines that start with the text (case-insensitive) `point:`. For > example, the following are all valid points: > > * `#### POINT: This is a point` > * `* point: This is a point` > * `+ Point: This is a point` > * `- pOINT: This is a point` > * `point: This is a **point**` > > Note that any markdown used in the point text will also be propagated > into the topic summaries. > > **Outcomes** > > Outcomes define the decisions or resolutions of a discussion. Once > outcomes are defined, sub-topics and points are collapsed > underneath the outcomes. > > Outcomes are declared in a similar manner as points: > > * `#### OUTCOME: This is an outcome` > * `* outcome: This is an outcome` > * `+ Outcome: This is an outcome` > * `- oUTCOME: This is an outcome` > * `outcome: This is an outcome` > > Note that multiple outcomes may be declared for each topic. > > **Topics** > > Topics can be stand-alone and contained within an issuable (epic, > issue, MR), or can be inline. > > Inline topics are defined by creating a new thread (discussion) > where the first line of the first comment is a heading that starts > with (case-insensitive) `topic:`. For example, the following are all > valid topics: > > * `# Topic: Inline discussion topic 1` > * `## TOPIC: **{+A Green, bolded topic+}**` > * `### tOpIc: Another topic` > > **Quick Actions** > > | Action | Description | > |-------------------------------|---------------------------------------------------------| > | `/discuss sub-topic TITLE` | Create an issue for a sub-topic. Does not work in epics | > | `/discuss link ISSUABLE-LINK` | Link an issuable as a child of this discussion | > > **Discussion-Size Indicators** > > The relative size of the discussion occurring within a topic > and its sub-topics is indicated via braille dots. > > More dots means that more points or sub-topics exist within a > given topic. > > Examples: > > * TOPIC `⣿⣿⡆` A large discussion occurred here > * TOPIC `⣇ ` A smaller discussion occurred here --- </details> Last updated by [this job](https://gitlab.com/gitlab-org/secure/pocs/discoto-runner/-/jobs/1534373358) <ul><li><details><summary>TOPIC <code title='Relative Number of Notes'>⡀ </code> <code title='Relative Number of Actions'> </code> Usability impact of non-prettified security report in the UI https://gitlab.com/groups/gitlab-org/-/epics/6602#note_659487796</summary><ul></ul></details></li><li><details><summary>TOPIC <code title='Relative Number of Notes'>⡄ </code> <code title='Relative Number of Actions'> </code> post-analyzer approach https://gitlab.com/groups/gitlab-org/-/epics/6602#note_659487816</summary><ul></ul></details></li></ul> <!-- end-discoto-summary --> <!-- start-discoto-topic-settings --><details> <summary>Discoto Settings</summary> <br/> ```yaml --- summary: max_items: -1 sort_by: created sort_direction: ascending ``` See the [settings schema](https://gitlab.com/gitlab-org/secure/pocs/discussion-automation#settings-schema) for details. </details> <!-- end-discoto-topic-settings -->