Category Maturity Scorecard - Secure:Dynamic Analysis FY22-Q1 - Verifying "Viable" status
Evaluation Results
The DAST Category Maturity Scorecard (CMS) to verify Viable maturity resulted in a rating of 3.5 (C), which is towards the top of the Viable range. This score officially verifies DAST's maturity as Viable and keeps it on track to advance to Complete maturity by 2022-04-30. View the Dovetail research insights to learn more about the study
👉 Dovetail research insights 👈
Additional resources
- Research issue: ux-research#1358 (closed)
- Script: https://docs.google.com/document/d/1Crfi90TlqZbUQm7dOtjdwtFbiMoVuXIwsix3gLUMTUw/edit
- Score calculation spreadsheet: https://docs.google.com/spreadsheets/d/1E3RPPoKC419OQz6Q7Pb82lwg_OvtYsJ1AT7ymi_LoNs/edit#gid=0
- Recommendations:
Category Maturity Scorecard Checklist
Learn more about Category Maturity Scorecards
-
Review the Category Maturity Scorecard handbook page and follow the process as described. Reach out to the UX Researcher for your stage if you have questions. -
Document the results of each JTBD scenario using this template -
Add links for each participants' session recordings. Consider downloading the session videos from Zoom, and uploading them to the shared Google UX Research folder in the appropriate section/stage/project folder. -
If the participant has not granted permission to share the recording publicly, ensure the sharing settings are set to GitLab-only. -
If needed, create a recommendation issue for these sessions.
CM Scorecard Results Template
Job to be done A
- When I am ready to release changes into production, I want to verify it is safe to release, So that I can release the changes responsibly.
Job to be done B
- When I am assessing the security of my application in production, I want to know whether my app is currently vulnerable, So that I can address detected business-critical vulnerabilities
Scenarios
Scenario 1
- Prompt - Your team has just completed work on a complex new feature for your application. The code for these changes has already been tested, but now you need to verify that the staging site is secure before you release the new changes to your production site. How would you do that?
- Average UMUX Lite score for capabilities - 4.33
- Average UMUX Lite score for ease of use - 4.00
- How many participants were successful at the task - 6/6
- How many participants failed the task - 0/6
- Total number of errors each participant encountered while attempting to complete the task/scenario - 2
Participant Number | Successful | Failed | Number of Errors Encountered |
---|---|---|---|
P3 | 1 | ||
P5 | 0 | ||
P6 | 1 | ||
P7 | 0 | ||
P8 | 0 | ||
P10 | 0 |
Scenario 2
- Prompt - Your development team is getting ready to release a new feature. There’s an open merge request containing the new code, and you’d like to test it thoroughly before it gets merged into your project’s main branch. You specifically want to check for cross-site scripting (XSS) and SQL injections, since these came up in a recent security assessment. Update the project's pipeline to check new merge requests for run-time vulnerabilities.
- Average UMUX Lite score for capabilities - 3.25
- Average UMUX Lite score for ease of use - 2.75
- How many participants were successful at the task - 4/5
- How many participants failed the task - 1/5
- Total number of errors each participant encountered while attempting to complete the task/scenario - 17
Participant Number | Successful | Failed | Number of Errors Encountered |
---|---|---|---|
P1 | 2 | ||
P2 | 6 | ||
P4 | 2 | ||
P9 | 2 | ||
P11 | 5 |
{{continue with as many scenarios as you used}}
Participants
Participant 1
- Role: Senior backend engineer, Create: editor
- Top 3 tasks/responsibilities
- Doing a lot of frontend and backend work on the editor team
- Previous/current GitLab usage: Daily use
- Session recording: https://dovetailapp.com/data/4J9vsoGk8fef06oi2MzLGD
Participant 2
- Role: Staff backend engineer, Sharding
- Top 3 tasks/responsibilities
- Backend development to make GitLab's load balancer support multiple databases
- Previous/current GitLab usage: 6+ years, currently uses daily
- Session recording: https://dovetailapp.com/data/4NxUgungasZLhuNi7WnO3Z
Participant 3
- Role: Senior security engineer, Security automation
- Top 3 tasks/responsibilities
- designing and maintaining internal security automation tools
- running automation (using the tools built by their team)
- Previous/current GitLab usage: 3+ years, currently uses daily
- Session recording: https://dovetailapp.com/data/5wYduJyf6LXzpGokoJeBwT
Participant 4
- Role: Backend engineer, Verify: Runner
- Top 3 tasks/responsibilities
- infrastructure reliability
- removing single points of failure
- uniformity and accessibility (others can contribute) to infrastructure
- Previous/current GitLab usage: 6+ years, currently uses daily
- Session recording: https://dovetailapp.com/data/6GCWiK7rVmI5DlFoRgwT9p
Participant 5
- Role: Senior security engineer, Application security
- Top 3 tasks/responsibilities
- review and manage security issues
- assess new features for security concerns
- Previous/current GitLab usage: 3 years, currently uses daily
- Session recording: https://dovetailapp.com/data/47tAmMTHd0Ug35qx9MYSO0
Participant 6
- Role: Senior security engineer, Red team
- Top 3 tasks/responsibilities
- designing attack operations
- model attack operations (threat modeling)
- smaller, targeted attacks on particular areas of the application
- Previous/current GitLab usage: 2+ years of frequent use, currently uses daily
- Session recording: https://dovetailapp.com/data/3xWiPCCHRYVdwprNiI1OTb
Participant 7
- Role: Senior security engineer, Application security
- Top 3 tasks/responsibilities
- vulnerability management
- application security reviews
- security reviews for new issues and proposals
- Previous/current GitLab usage: 2+ years of frequent use, currently uses daily
- Session recording: https://dovetailapp.com/data/6qOPOMTDll6i8dHzx5iQ9t
Participant 8
- Role: Security engineer, Red team
- Top 3 tasks/responsibilities
- identify and communicate risks
- emulate attackers
- work on defensive security measures
- Previous/current GitLab usage: 4+ years, currently uses daily
- Session recording: https://dovetailapp.com/data/6BUpSdPaq9D74h2TxjLEcn
Participant 9
- Role: Backend engineer, Scalability & infrastructure
- Top 3 tasks/responsibilities
- observability and reliability of infrastructure
- GitLab maintainer
- Incident review
- Previous/current GitLab usage: 4+ years, currently uses daily
- Session recording: https://dovetailapp.com/data/6k1ItrmUyA6dIUbudkdbXs
Participant 10
- Role: Staff security engineer, Security research
- Top 3 tasks/responsibilities
- securing gitlab.com (the product)
- developing security tools
- contributing to the larger security community (evangelizing GitLab as a security expert)
- Previous/current GitLab usage: 2+ years of regular use plus some previous experience, currently uses daily
- Session recording: https://dovetailapp.com/data/5II4lsJGrP32llXKl2adK8
Participant 11
- Role: Senior backend engineer, Monitor
- Top 3 tasks/responsibilities
- greenfield development
- writing & reviewing code
- planning and scoping new features
- Previous/current GitLab usage: 2.5 years, currently uses daily
- Session recording: https://dovetailapp.com/data/3atGsbKDfVYDotwVRkXmul
Edited by Valerie Karnes