Associate Build with DAST Site and Scanner profile [RUN ALL RSPEC] [RUN AS-IF-FOSS] (!63362) · Merge requests · GitLab.org / GitLab

Philip Cunningham requested to merge philipcunningham-add-relationship-between-builds-and-dast-profiles-216514 into master Jun 07, 2021

What does this MR do?

adds an association between ci_builds and dast_site_profiles + dast_scanner_profiles in order to deliver calculated variables to ci_builds.

Example

when you specify a site or scanner profile in the yaml config, we will look up those records in the database by name and add the calculated variables to the build using DastSiteProfile#ci_variables and DastScannerProfile#ci_variables.

dast:
  stage: dast
  dast_configuration:
    site_profile: "site-profile-name"
    scanner_profile: "scanner-profile-name"
  script:
   - echo

the merge request that adds that adds the job-level keyword is here and the merge request that seeds the build is here.

Context

with dast we have the concept of on-demand scans, which essentially entails storing dast config in the database (dast_scanner_profiles and dast_site_profiles) and triggering scans via GraphQL on a manual basis. the proposal is to reference those database entities by name in the .gitlab-ci.yml and seed their configuration into pipelines triggered via the regular ci/cd process.

Why?

whilst refining this feature we got feedback that indicate the modelling we already had in place (between dast_profiles and ci_pipelines) would fit with what we were trying to achieve. it was therefore necessary to introduce a different relation to better capture that there might be many dast builds for a given pipeline.

Related Issue(s)

Use DAST Scan profiles from database

Related Merge Requests

Database

Migration

rails db:migrate:up VERSION=20210604032738 && \
  rails db:migrate:up VERSION=20210604034158 && \
  rails db:migrate:up VERSION=20210604034354 && \
  rails db:migrate:up VERSION=20210604051330 && \
  rails db:migrate:up VERSION=20210604051742 && \
  rails db:migrate:up VERSION=20210604051917

== 20210604032738 CreateDastSiteProfilesBuilds: migrating =====================
-- create_table(:dast_site_profiles_builds, {:primary_key=>[:dast_site_profile_id, :ci_build_id], :comment=>"{\"owner\":\"group::dynamic analysis\",\"description\":\"Join table between DAST Site Profiles and CI Builds\"}"})
   -> 0.0135s
== 20210604032738 CreateDastSiteProfilesBuilds: migrated (0.0137s) ============

== 20210604034158 AddCiBuildIdFkToDastSiteProfilesBuilds: migrating ===========
-- transaction_open?()
   -> 0.0000s
-- foreign_keys(:dast_site_profiles_builds)
   -> 0.0048s
-- execute("ALTER TABLE dast_site_profiles_builds\nADD CONSTRAINT fk_a325505e99\nFOREIGN KEY (ci_build_id)\nREFERENCES ci_builds (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0041s
-- execute("SET statement_timeout TO 0")
   -> 0.0014s
-- execute("ALTER TABLE dast_site_profiles_builds VALIDATE CONSTRAINT fk_a325505e99;")
   -> 0.0067s
-- execute("RESET ALL")
   -> 0.0012s
== 20210604034158 AddCiBuildIdFkToDastSiteProfilesBuilds: migrated (0.0358s) ==

== 20210604034354 AddDastSiteProfileIdFkToDastSiteProfilesBuilds: migrating ===
-- transaction_open?()
   -> 0.0000s
-- foreign_keys(:dast_site_profiles_builds)
   -> 0.0039s
-- execute("ALTER TABLE dast_site_profiles_builds\nADD CONSTRAINT fk_94e80df60e\nFOREIGN KEY (dast_site_profile_id)\nREFERENCES dast_site_profiles (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0021s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- execute("ALTER TABLE dast_site_profiles_builds VALIDATE CONSTRAINT fk_94e80df60e;")
   -> 0.0047s
-- execute("RESET ALL")
   -> 0.0006s
== 20210604034354 AddDastSiteProfileIdFkToDastSiteProfilesBuilds: migrated (0.0193s)

== 20210604051330 CreateDastScannerProfilesBuilds: migrating ==================
-- create_table(:dast_scanner_profiles_builds, {:primary_key=>[:dast_scanner_profile_id, :ci_build_id], :comment=>"{\"owner\":\"group::dynamic analysis\",\"description\":\"Join table between DAST Scanner Profiles and CI Builds\"}"})
   -> 0.0154s
== 20210604051330 CreateDastScannerProfilesBuilds: migrated (0.0156s) =========

== 20210604051742 AddCiBuildIdFkToDastScannerProfilesBuilds: migrating ========
-- transaction_open?()
   -> 0.0000s
-- foreign_keys(:dast_scanner_profiles_builds)
   -> 0.0047s
-- execute("ALTER TABLE dast_scanner_profiles_builds\nADD CONSTRAINT fk_e4c49200f8\nFOREIGN KEY (ci_build_id)\nREFERENCES ci_builds (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0022s
-- execute("SET statement_timeout TO 0")
   -> 0.0009s
-- execute("ALTER TABLE dast_scanner_profiles_builds VALIDATE CONSTRAINT fk_e4c49200f8;")
   -> 0.0061s
-- execute("RESET ALL")
   -> 0.0010s
== 20210604051742 AddCiBuildIdFkToDastScannerProfilesBuilds: migrated (0.0253s)

== 20210604051917 AddDastScannerProfileIdFkToDastScannerProfilesBuilds: migrating
-- transaction_open?()
   -> 0.0000s
-- foreign_keys(:dast_scanner_profiles_builds)
   -> 0.0047s
-- execute("ALTER TABLE dast_scanner_profiles_builds\nADD CONSTRAINT fk_5d46286ad3\nFOREIGN KEY (dast_scanner_profile_id)\nREFERENCES dast_scanner_profiles (id)\nON DELETE CASCADE\nNOT VALID;\n")
   -> 0.0023s
-- execute("SET statement_timeout TO 0")
   -> 0.0009s
-- execute("ALTER TABLE dast_scanner_profiles_builds VALIDATE CONSTRAINT fk_5d46286ad3;")
   -> 0.0029s
-- execute("RESET ALL")
   -> 0.0006s
== 20210604051917 AddDastScannerProfileIdFkToDastScannerProfilesBuilds: migrated (0.0226s)

rails db:migrate:down VERSION=20210604051917 && \
  rails db:migrate:down VERSION=20210604051742 && \
  rails db:migrate:down VERSION=20210604051330 && \
  rails db:migrate:down VERSION=20210604034354 && \
  rails db:migrate:down VERSION=20210604034158 && \
  rails db:migrate:down VERSION=20210604032738

== 20210604051917 AddDastScannerProfileIdFkToDastScannerProfilesBuilds: reverting
-- remove_foreign_key(:dast_scanner_profiles_builds, {:column=>:dast_scanner_profile_id})
   -> 0.0066s
== 20210604051917 AddDastScannerProfileIdFkToDastScannerProfilesBuilds: reverted (0.0142s)

== 20210604051742 AddCiBuildIdFkToDastScannerProfilesBuilds: reverting ========
-- remove_foreign_key(:dast_scanner_profiles_builds, {:column=>:ci_build_id})
   -> 0.0058s
== 20210604051742 AddCiBuildIdFkToDastScannerProfilesBuilds: reverted (0.0149s)

== 20210604051330 CreateDastScannerProfilesBuilds: reverting ==================
-- drop_table(:dast_scanner_profiles_builds)
   -> 0.0035s
== 20210604051330 CreateDastScannerProfilesBuilds: reverted (0.0036s) =========

== 20210604034354 AddDastSiteProfileIdFkToDastSiteProfilesBuilds: reverting ===
-- remove_foreign_key(:dast_site_profiles_builds, {:column=>:dast_site_profile_id})
   -> 0.0061s
== 20210604034354 AddDastSiteProfileIdFkToDastSiteProfilesBuilds: reverted (0.0146s)

== 20210604034158 AddCiBuildIdFkToDastSiteProfilesBuilds: reverting ===========
-- remove_foreign_key(:dast_site_profiles_builds, {:column=>:ci_build_id})
   -> 0.0061s
== 20210604034158 AddCiBuildIdFkToDastSiteProfilesBuilds: reverted (0.0149s) ==

== 20210604032738 CreateDastSiteProfilesBuilds: reverting =====================
-- drop_table(:dast_site_profiles_builds)
   -> 0.0054s
== 20210604032738 CreateDastSiteProfilesBuilds: reverted (0.0054s) ============

Does this MR meet the acceptance criteria?

Conformity

I have included a changelog entry, or it's not needed. (Does this MR need a changelog?)
I have added/updated documentation, or it's not needed. (Is documentation required?)
I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
I have self-reviewed this MR per code review guidelines.
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
I have followed the style guides.
This change is backwards compatible across updates, or this does not apply.

Availability and Testing

I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)
I have tested this MR in all supported browsers, or it's not needed.
I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

Label as security and @ mention @gitlab-com/gl-security/appsec
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
Security reports checked/validated by a reviewer from the AppSec team

Edited Jun 18, 2021 by Shinya Maeda

Associate Build with DAST Site and Scanner profile [RUN ALL RSPEC] [RUN AS-IF-FOSS]