Discussion: Direction of api fuzz configuration, using UI->Database or UI-> MR to change yml file
Background Info
Scope
We are discussing the enable/disable/fine-tune configuration for fuzz testing in the context of CI/CD. The on-demand scan is not part of the discussion here, mentioning it is used as a reference for discussion, but we are discussing the possible product direction for it. Here the direction is ONLY for the configuration in the CI/CD part.
Note: currently we are only implementing API fuzz testing configuration, but gl-fuzz might face the same problem
Current solution plan:
Users can configure an API fuzz test using a UI form, and after the user does their changes, we will generate code snippet and user needs to copy/paste to the right place themselves
Discussion Point - two future directions
Direction 1. UI -> MR -> .yml
Explanation: User can change something in the UI, potentially everything in the future, including profiles, route, checks, assertions. It is a complicated api-fuzz.yml content itself without considering other stuff inside GitLab-ci.yml file.
UI will be a huge help for the user to process the info rather than trying to parse the complicated structure in the yml file manually. But in the end, the user is still seeing the whole file they are changing. So there might be potential help if we can split the settings into different yml files.
- Potential idea for this direction
- 1.1 all configuration code is part of the gitlab-ci.yml file (Current)
- 1.2 all configuration code is in one fuzz.yml file
- 1.3 all configuration code is in several yml files based on functionality, such as profile-fuzz.yml checks-fuzz.yml files
Direction2.1 Directly save changes: either in the database or in repo
Explanation: The user can change something in the UI, and click on the Save button. The result will be to sync to the database, ideally, sync to yml file too or we inform the user to update the yml file or not; another way around this is the same as if the user updates yml, it also syncs to the database, and we inform the user in the UI it has been changed and by which MR. So the database is always the SSOT; the last option is that the user can't edit the details in yml, every configuration is via UI and database.
For the user, this could be an easier solution, what the user changes in the UI and saved is what is exactly affected in the system. The user can create a draft if needed, ask someone else to review and finally save it. Ideally, no matter if the user updates the UI or yml file, they all sync to the database, so there is no hierarchy in who overwrote who, but if not possible, we can have the following structural changes to help? Just some "wild" ideas I'm thinking about:
- Potential Solutions for this direction
- 2.1 Everything is in one or several xxx.fuzz.yml files and have 1:1 mappings to the database
- 2.2 SSOT is in the database, inform users both in UI where the last update on the database was done
- 2.3 SSOT is in the UI, user can't use yml file to enable/disable security scan
- 2.4 Directly save changes in repo ./gitlab, a separate yml file, don't let user create an MR, but have a separate yml file which user has "read-only" right
Here I would like to discuss the two directions in general, these detailed potential solutions are just some ideas as a reference to asset discussions.
The main goal is to figure out what the best user experience could be for these two general directions and if we chose one over the other, what are the compromises we are taking and what efforts will we need to put into it.
User needs
situations description: who does what
Current user habit (Assumptions, not validated yet):
- Users could be very programmatic oriented, which means they might more comfortable using code, UI might be a secondary choice
- User could be familiar with UI setting especially those who tune configuration in a very advance level, such as turn on/off for check/assertions or detail setting for each check/assertions.
- User copy/paste settings a lot, no matter which direction
- User might need backend functionality such as embed debug/verbose flags (@stkerr, could you explain more here, how to rephrase No.iii points here? )
- Team structure possibility 1: Everyone has the same right, everyone can edit, save configurations
- Team structure possibility 2: In one team there are different roles, everyone can edit it, but need to be reviewed by a person to confirm the changes
- Cameron could also be a target user, who ensure compliance since he will have an artefact of some sort to save and say “release X was tested with this specific configuration” ( Additional target user besides Sam-Security-Analyst and Sasha-Developer)
Note, we don't know which situation is the most common one
Prons | Crons | |
---|---|---|
1. UI -> MR -> *.yml | All change history is logged. It is easier for collaboration if there is an approval process needed, yml file is easy Easier when the backend support needed Might be preferred by programmatic oritented user |
After user saves changes, the yml file might still be big, not easy to spot what has been edited in yml file directly When there are several MRs, in the UI we can't communicate easily what has been changed in which MR Potential bugs when yml file is big |
2. Directly save changes: either in the database or in repo | It is easy to update changes in UI and directly saving is a common practice in other security tools easy learn/adapt for all type of users at all levels potential history log can be designed in UI |
If we allow the user to use both yml file and UI, how do we sync them and how do we communicate this to the user... this is challenging; if we remove the ability user can use yml file, we don't have this problem, we just need to make it clear! |
Tech ability
Stylesheet: https://docs.google.com/spreadsheets/d/19EtkJcbabgkNuyMVsvU4INyqmoOrmik520B2y0vkDAM/edit?usp=sharing
Time needed | Potential problems | Other | |
---|---|---|---|
1.1 UI -> MR -> ci-gitlab.yml | |||
1.2 UI -> MR -> api-fuzz.yml | |||
1.3 UI -> MR -> api-fuzz1.yml ,api-fuzz2.yml | |||
2.1 UI sync with yml | |||
2.2 Database is SSOT, user can udpate both yml and UI | |||
2.3 UI only | |||
2.4 Directly save changes in repo ./gitlab |