Add scan duration check

What does this MR do?

This MR adds a scan duration check to the qa template for sast. It also adds a new variable SCAN_DURATION_MARGIN_PERCENT for flexibility in alerting on scan duration that exceeds MAX_SCAN_DURATION_SECONDS. Because the job is added in the qa template, this change will allow the testing of scan duration per downstream job.

If $MAX_SCAN_DURATION_SECOND is not set this job will not run.

Here is an example run in gosec: https://gitlab.com/gitlab-org/security-products/analyzers/gosec/-/pipelines/215535529

The ci file has 4 jobs against tests/go-modules with different combinations of variables set: https://gitlab.com/gitlab-org/security-products/analyzers/gosec/-/blob/196697-test-scan-duration-script-sast/.gitlab-ci.yml

1. MAX_SCAN_DURATION_SECONDS is not set: pass and report on SCAN_DURATION
2. MAX_SCAN_DURATION_SECONDS > SCAN_DURATION: pass
3. MAX_SCAN_DURATION_SECONDS < SCAN_DURATION: fail
4. MAX_SCAN_DURATION_SECONDS + SCAN_DURATION_MARGIN_PERCENT < SCAN_DURATION: pass

The downstream test (go-modules) was modified to load this branch's qa template: https://gitlab.com/gitlab-org/security-products/ci-templates/-/tree/196697-add-scan-duration-check-to-qa-tpl

Once this MR is merged, the process will be to set the variable to 0 and trigger each analyzer so as to get a more precise reading for each downstream project.

What are the relevant issue numbers?

gitlab-org/gitlab#196697 (closed)

Does this MR meet the acceptance criteria?

• Tests added for this feature/bug