Store DAST scan results into the database

What does this MR do?

Add DAST parser and store DAST vulnerabilities in the database.

Updated: this discussion about location_fingerprints. https://gitlab.com/gitlab-org/gitlab-ee/issues/7062#note_131695170 is finished. It was decided that we flatten vulnerabilities, and for each instance in the report should be one occurrence in the database.

What are the relevant issue numbers?

#7062 (closed)

Does this MR meet the acceptance criteria?

• Changelog entry added, if necessary
• Documentation created/updated via this MR
• Tests added for this feature/bug
• Tested in all supported browsers
• Conforms to the code review guidelines
• Conforms to the merge request performance guidelines
• Conforms to the style guides
• Conforms to the database guides
• Link to e2e tests MR added if this MR has Requires e2e tests label. See the Test Planning Process.
• EE specific content should be in the top level /ee folder
• For a paid feature, have we considered GitLab.com plans, how it works for groups, and is there a design for promoting it to users who aren't on the correct plan?
• Security reports checked/validated by reviewer

Closes #7062 (closed)

changed milestone to %11.8

added Category:DAST Deliverable GitLab Ultimate backend devopssecure ~2975006 + 1 deleted label

added 1 commit

• 4e706ef5 - Add newline after self review

Compare with previous version

@gonzoyumo could you please review this MR? It's not ready for a merge because there is no decision about location fingerprint yet. However, your comments will be valuable for me at this stage.

@gonzoyumo What about using the pluginid as the primary identifier here?

See https://github.com/zaproxy/zaproxy/blob/w2019-01-14/docs/scanners.md

added 1 commit

• 10a42455 - Fix rubocop issues

Compare with previous version

@brytannia thanks, I left some comments. I'm also looking forward discussion to resolve https://gitlab.com/gitlab-org/gitlab-ee/issues/7062#note_131695170.

assigned to @brytannia

added 310 commits

• 10a42455...9d9f254a - 293 commits from branch master
• 296de629 - Add dast feature flag
• d144bb21 - Add dast parser
• eadf6102 - Add DAST to licensed features
• 562f2349 - Add DAST to parsers
• 4605bb8f - Save dast vulnerability to db
• 379963fa - Prettify dast parser
• 3a854164 - Reformat parser due to new info
• 0bb21bf1 - Add test for dast feature flag
• 8a7de274 - Fix spelling mistake
• 8d0ba1ef - Fix issues after testing
• b49106de - Add newline after self review
• b0a831ab - Fix rubocop issues
• 7d023068 - Add changelog entry
• 6148e980 - Address comments in review
• aa53b7f7 - Add test for FF default cases
• 98fc4d43 - Add new identifier and change version
• 3ab24fc6 - Update levels according to spec

Compare with previous version

added 130 commits

• 3ab24fc6...15d67855 - 113 commits from branch master
• 0deef7f8 - Add dast feature flag
• 6f186be6 - Add dast parser
• 125484db - Add DAST to licensed features
• f7d2131b - Add DAST to parsers
• 7afd0c4e - Save dast vulnerability to db
• b022ebcf - Prettify dast parser
• a74a293e - Reformat parser due to new info
• 307e5521 - Add test for dast feature flag
• a7da137f - Fix spelling mistake
• 0a074ea8 - Fix issues after testing
• 4d925cfa - Add newline after self review
• fb751c51 - Fix rubocop issues
• b8cd44d2 - Add changelog entry
• b1177d09 - Address comments in review
• b1cb9cd4 - Add test for FF default cases
• 9df644e9 - Add new identifier and change version
• af514107 - Update levels according to spec

Compare with previous version

added 1 commit

• 95f0219d - Add location fingerprint implementation

Compare with previous version

unmarked as a Work In Progress

changed the description

assigned to @gonzoyumo

@gonzoyumo I addressed all your comments and implemented location fingerprints. Could you look at​ this MR again, please?

@brytannia thanks. I have some doubts on few things that makes me think this MR has deep implications on our model and report format. For instance, I think the values we currently have for confidence and severity are not matching the all needs of DAST vulnerabilities. The Location property also mismatches and misses properties in our common format. Technically we are not constrained to follow the common format but this might cause us big troubles in the future.

I also still have some hope that we all agree on flattening the 'instances' array into several vulnerabilities occurrences :)

assigned to @brytannia

added 2 commits

• c04e70fb - Remove redundant test cases
• 7cfbb90f - Flatten dast vulnerabilities

Compare with previous version

@gonzoyumo I've flattened 'instances' into separate occurrences as it was discussed previously. Please, review

assigned to @gonzoyumo

@brytannia thanks, I left few minor comments. I think you can resolve them by yourself once fixed and assign a reviewer.

assigned to @brytannia

added 952 commits

• 7cfbb90f...15bf2c94 - 951 commits from branch master
• 4d5c311a - Add DAST parser

Compare with previous version

@stanhu could you please review this MR?

You were a reviewer for https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8797, and both issues are pretty similar.

assigned to @stanhu

assigned to @tkuah

I'm reassigning this to @tkuah for now because I am fully focused on a customer issue at the moment.