Add scan duration check to Secret Detection template

What does this MR do?

This MR adds a scan duration check to the qa template for secret detection. It also adds a new variable SCAN_DURATION_MARGIN_PERCENT for flexibility in alerting on scan duration that exceeds MAX_SCAN_DURATION_SECONDS. This job allows more precise testing of scan duration by reading the start and end times of the report. And it allows testing to become more granular by allowing a test for each downstream project.

I added an example run to secrets analyzer: https://gitlab.com/gitlab-org/security-products/analyzers/secrets/-/pipelines/227746652

The 3 new downstream jobs are only there for review in this MR (the branch will not be merged) showing:

• if MAX_SCAN_DURATION_SECONDS is not set: https://gitlab.com/gitlab-org/security-products/tests/secrets/-/pipelines/227747792
• if MAX_SCAN_DURATION_SECONDS is exceeded: https://gitlab.com/gitlab-org/security-products/tests/secrets/-/pipelines/227747793 (this one is supposed to fail but secrets just runs too fast 😂) you can see an example failing scan duration check for the sast job https://gitlab.com/gitlab-org/security-products/tests/go-modules/-/pipelines/229710526
• if MAX_SCAN_DURATION_SECONDS is not exceeded: https://gitlab.com/gitlab-org/security-products/tests/secrets/-/pipelines/227747795

The analyzer branch points to the downstream branch for tests/secrets which in turn points to this branch in ci-templates.

Once this MR is merged, the process will be to set the variable to 0 (in a test branch) and trigger a pipeline in the SD analyzer so as to get a more precise reading for the downstream project.

What are the relevant issue numbers?

gitlab-org/gitlab#196697 (closed)

Does this MR meet the acceptance criteria?

• Tests added for this feature/bug