Move Secure test projects to Secure analyzer projects

Proposal

Move the Secure test projects to Secure analyzer projects, for ease of maintenance. This can probably be achieved using dynamic child pipelines - to be proven in a PoC.

Pros:

  • community contributors can run full QA for the analyzer projects they contribute too
  • new test projects can be introduced in feature branches of the analyzer project; the test project is reviewed when reviewing the feature, and it's added when merging the feature branch; it's a one-step process, as opposed to the current back and forth b/w the analyzer project and the test projects
  • similarly, new variants to the existing test projects are introduced in branches of the analyzer projects that change or fix the behavior
  • there's no need to disable other analyzers that are not being tested; there's no need for SAST_DEFAULT_ANALYZERS, DS_DEFAULT_ANALYZERS, SAST_DISABLED, DS_DISABLED, etc.
  • when updating a test project used by an analyzer, we no longer take the risk of breaking QA for another analyzer; there's no unintended side-effect

/cc @willmeek @theoretick @gonzoyumo

Auto-Summary 🤖

Discoto Usage

Points

Discussion points are declared by headings, list items, and single lines that start with the text (case-insensitive) point:. For example, the following are all valid points:

  • #### POINT: This is a point
  • * point: This is a point
  • + Point: This is a point
  • - pOINT: This is a point
  • point: This is a **point**

Note that any markdown used in the point text will also be propagated into the topic summaries.

Outcomes

Outcomes define the decisions or resolutions of a discussion. Once outcomes are defined, sub-topics and points are collapsed underneath the outcomes.

Outcomes are declared in a similar manner as points:

  • #### OUTCOME: This is an outcome
  • * outcome: This is an outcome
  • + Outcome: This is an outcome
  • - oUTCOME: This is an outcome
  • outcome: This is an outcome

Note that multiple outcomes may be declared for each topic.

Topics

Topics can be stand-alone and contained within an issuable (epic, issue, MR), or can be inline.

Inline topics are defined by creating a new thread (discussion) where the first line of the first comment is a heading that starts with (case-insensitive) topic:. For example, the following are all valid topics:

  • # Topic: Inline discussion topic 1
  • ## TOPIC: **{+A Green, bolded topic+}**
  • ### tOpIc: Another topic

Quick Actions

Action Description
/discuss sub-topic TITLE Create an issue for a sub-topic. Does not work in epics
/discuss link ISSUABLE-LINK Link an issuable as a child of this discussion

Discussion-Size Indicators

The relative size of the discussion occurring within a topic and its sub-topics is indicated via braille dots.

More dots means that more points or sub-topics exist within a given topic.

Examples:

  • TOPIC ⣿⣿⡆ A large discussion occurred here
  • TOPIC ⣇ A smaller discussion occurred here

Last updated by this job

  • TOPIC ⣿⡄ ⣼ Move fixtures to analyzer projects (security-products/analyzers) https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_490022538
    • remove side effects of changing fixtures on other feature categories, giving more autonomy and velocity to each team https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_490026439
    • duplicate similar projects setup/environments between the different feature categories https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_490026439
    • can change a test project in a MR of the analyzer w/o breaking QA (in the scenario where the latest release of the analyzer doesn't behave the same; we've been there https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_499292168
    • can introduce a test project in a MR of the analyzer w/o breaking QA cc @willmeek we've been there https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_499292168
    • self contained projects, which is good for readability https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_499292168
    • no external dependencies, which makes this community contribution-ready https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_499292168
  • TOPIC ⡇ ⢠ Keep fixtures in separate projects (security-products/tests) https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_490023296
    • shared test projects https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_490136114
    • side effects between teams sharing a project. Pipeline failure can affect other teams while error is unrelated to their product area. https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_490149206
  • TOPIC ⣷ ⢸ Remove QA jobs from test projects https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_494382731
    • better separation of concerns; test projects are only fixtures, and they're not responsible for QA anymore https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_499296835
    • CI config used for QA is self-contained; it lives in the analyzer projects, and there's no need to jump back and forth b/w projects to make sense of it https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_499296835
    • changes to the way we do QA can be changed w/o changing test projects; for instance, we can experiment with changes to `qa-sast.yml` without changing any test project https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_499296835
    • intermediary step before moving the test projects to the analyzer projects https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_499296835
  • TOPIC ⡀ ⣸ Brainstorming session summary https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_505277293
    • PoC should move forward but using sub-modules and a more complex project to test capability https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_505277293
    • our analyzers bleed many testing levels and we should better separate functional from integration tests https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_505277293
    • the current PoC still does not help us with testing locally, but gets us closer https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_505277293
    • there are concerns about the assumptions our analyzer environment has that will be problematic in a single project; i.e. checking `CI_PROJECT_DIR` for the project root -- would this refer to the analyzer now or the test project? https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_505277293
    • performance/time-cost considerations between executing within a job vs a child pipeline https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_505277293
  • TOPIC ⣷ ⣼ Move test projects to branches of the analyzer projects https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_603111498
    • analyzer projects come with all the test projects they depend on https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_603111498
    • community contributors can run job integration tests https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_603111498
    • pipelines can be dynamically generated https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_603111498
    • real job integration tests https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_603111498
    • test projects can't be updated in a bug or feature branch https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_603111498
    • test projects are repeated across analyzer projects https://gitlab.com/gitlab-org/gitlab/-/issues/297361#note_603111498
Discoto Settings 
---
summary:
  max_items: -1
  sort_by: created
  sort_direction: ascending

See the settings schema for details.

Edited by 🤖 GitLab Bot 🤖