Experience Recommendations - Secure FY20-Q3 - Security Dashboard

changed the description

Hello! Based on the part 1 experience audit, I prioritized these items as they're front-end focused and aim to make visible the most important data to the user: the vulnerabilities table.

Proposed next steps: get feedback from the team, iterate further on these designs, and then work with @leipert team on capacity planning.

@andyvolpe @vkarnes @leipert @NicoleSchwartz Would love to hear any feedback, thoughts, suggestions on the proposal, and/or next steps - Thank you!

added to epic &1467 (closed)

@andyvolpe I noticed the screenshot below is displayed on a project level security dashboard. It links the branch and commits. Although I didn't see this section in the GDK (at the group level) - is this a standard display in production?

Having this data displayed is a good start toward mitigating problem C listed in the description. The vulnerabilities displayed in the table are results of the most recent security scan on the default branch. Maybe could be helpful to make explicit to user by adding info text like: "Last updated [date of most recent security scan] link to branch". What do you think?

@kmann This section is only available at the project level. At the group level, we are aggregating all the results from each project that has run (a pipeline on master). Since we could be dealing with 100+ projects with different pipelines running at different times, we opted not to show this information. This exact information is not as useful here as at the project level where users can dig into the pipeline report or the project dashboard and start resolving vulns.

There is an issue open around the problem you listed: https://gitlab.com/gitlab-org/gitlab-ee/issues/7521 It's meant to make users aware that their dashboard is not up-to-date. That might be a good starting point for this issue.

It’s unclear what time frame the table vulnerabilities are displaying

This is true, though we want to convey that this is a list of active vulnerabilities that currently exist in the default_branch aka (master_branch) as of the last pipeline that has run for each project.

Let me know if this helps.

Got it, thanks. In the proposed layout: the branch information could be displayed in the aside. Currently, it's placed at the top of the filters header, which pushes the vuls table data further down the page.

though we want to convey that this is a list of active vulnerabilities that currently exist in the default_branch aka (master_branch)

As a first step for consideration: link the documentation "When is the data updated?" or "Learn more about the data shown" or explicitly stating "The Security Dashboard displays information from the results of the most recent security scan on the default branch, which means that security scans are performed every time the branch is updated". Then it may help the user know why/how/when the data is being displayed and updated.

cc @dappelt

added Deliverable devopssecure + 1 deleted label

@kmann Would it be possible to expand the aside to better see the chart if needed?

Yes: ideally it would expand for larger viewing on desktop screens. When expanded, the layout could change as it would at lower screen sizes.

Expand Trigger	Layout Change (same as seen on smaller screen sizes)
	1) header 2) table 3) chart

@kmann @fcatteau

I was thinking more about the chart and I believe the chart layout in the aisle does a great job at letting the user know how well they are doing at keeping vulnerabilities to a minimum. The role of the chart here is a feedback loop for users of the dashboard so they are not operating in the dark. Without the chart, users who are managing this vulnerability list aren't able to see the big picture and how their work is impacting the group. It's a tiny bit of gamification where the goal is the lowest number possible.

Where the chart falls short is with its original intent. "To provide security Managers/Directors/CISO's a view of how their group has been performing over time". There isn't enough accompanying information to accurately summarize security KPIs at this time and detected vulnerabilities is just one metric that feeds into that.

As of now, we've been balancing these two needs in one chart and I believe it's (as it exists today) falling short at meeting either of those.

@kmann 's proposal does a great job a meeting the needs of the analysts who are the primary users of this area. I don't think the ability to expand the chart to see more data is adding anymore benefit to the analysts nor is it adding value to the CISO roles. Perhaps a way to download the chart and it's data would be a better feature for the CISO given this case.

Good points @andyvolpe. This goes back to us needing to understand the difference between snapshot views and reports.

@kmann

I noticed you removed the severity counters at the top of the dashboard. I think this information is valuable and might get lost if it's nested within the vuln chart. There might be room to address the appearance of these so they don't take up so much vertical space.

Thought?

Makes sense to explore that. My concern is it would be repetitive because in this case the severity count would be displayed on top of the table, in the line chart, and in the counter below the chart. Also, trying to avoid Problem B listed above - showing irrelevant counter data, when filters are applied. Do you think a counter in the filters could help? that is showing:

All Vulnerabilities (44)
Critical (20)
High (4)
Medium (14)
Low (6)

Another consideration: is leveraging sparklines to communicate our message. We want to show vulnerabilities over time,not necessarily how different vulnerability severity levels compare with each. In this case, the visual can be simplified to display the count and sparkline to the relevant vuls. Looks something like this:

This summarizes the data, in addition to telling a story of how the severity has changed over the last 30 days.

@kmann

leveraging sparklines to communicate our message. We want to show vulnerabilities over time,not necessarily how different vulnerability severity levels compare with each. In this case, the visual can be simplified to display the count and sparkline to the relevant vuls.

Yes! This is really hitting on the right idea of "how have we been doing over the past X days." Comparing severity levels against one another doesn't make sense, as you said, so I say we might have more flexibility in how we show change over time for each severity level. I really like the sparklines but I wonder if more exploration can be done with this new insight?

Design wise, I still think the amount of vulns present is taking too much of a back seat and it's hard for me to quickly assess how many of X vulns are present. Maybe those can be played up a bit more or presented in a different module? A combined label was explored in the past: https://gitlab.com/gitlab-org/gitlab-ee/issues/7226#note_94153536 maybe something like this or in a similar spirit might help.

@kmann first thing i think is do a dev review to see if current architecture is blocking this, and if yes what needs to change so that can be done first, then i think next step might be adjusting filters to work the way indicated, next the timeframe, next might be the floating graph?

@NicoleSchwartz Yes, agreed. @leipert who on the FE team should we sync with about this issue?

assigned to @kmann

Ideating on problem C in the description, and similar (https://gitlab.com/gitlab-org/gitlab-ee/issues/7521). I’m trying to clarify and define the following in this thread: update data we can display, update data that is useful for the user, and development constraint considerations. Currently, the data results in the group security dashboard is updated in two ways:

From the most recently updated default branch
From configured scheduled pipelines

We want the user to be aware of when the data was updated and what triggered the update to occur. Data that could be helpful to this understanding is:

Project name
Updated default branch
Scheduled scan
Timeframe the update occurred
Projects that are untested

Questions:

Do you have edits and/or suggestions to the data to display?
Are there constraints to displaying the above data in the group dashboard we need to consider?
Is there a time threshold that would make a project “out of date” or needing to be marked as similar?
Are there any actions we suspect the user may want to take, based on the data displayed?

@NicoleSchwartz @andyvolpe @gonzoyumo @plafoucriere @fcatteau (I'm not sure who can best answer these questions or who worked on the dashboard previously, CC the appropriate person if I didn't - thank you)

i) minimal step	ii) with proposed layout

i) As a first and minimal step, as mentioned above: we could link our documentation so the user at least has awareness of where/when the data is coming from.	ii) Other consideration: in the proposed layout in the description, the data could be displayed in the aside (section 3 seen in the layout description). The displayed data then aligns with any filters applied (for example only showing 3 of 10 projects, when filtered as such)

@kmann I like the direction! here are a few thoughts & answers to questions:

Do you have edits and/or suggestions to the data to display?

I am unsure if "Vulnerability Management" is a user-facing title at this time. I would keep the heading for the page the same as the page title to avoid any confusion. If we want to consider a new name for this area, we should also think about the IA, navigation, and breadcrumbs as well and have a conversation about dashboards vs lists in the Secure area.

Are there constraints to displaying the above data in the group dashboard we need to consider?

I believe there are. We are automatically importing security testing information from the project level. By doing this we don't allow the user to select which project they want to track security for and which they don't. This doesn't seem like a problem because we assume the user wants to only track projects they have set-up security scanning for. Problem is, our tools don't know that and we are returning a list of every project available with the group. This makes displaying when the last scan for X project tricky because we cannot discern between a project that wasn't scanned and needs to be and a project that the user doesn't want security scanning on.

Is there a time threshold that would make a project “out of date” or needing to be marked as similar?

Based on this issue: https://gitlab.com/gitlab-org/gitlab-ee/issues/7521 it should be 5 days or so.

Are there any actions we suspect the user may want to take, based on the data displayed?

Unsure. Maybe manually trigger a security scan if one has not been done or it's out-of-date for a project.

Thanks @andyvolpe! Moved this thread to https://gitlab.com/gitlab-org/gitlab-ee/issues/7521#note_185477023 as it's more closely related to that issue.

@kmann the vulnerabilities displayed in the group dashboard are associated with at least one pipeline (or several ones if vulnerability keeps being reported on each pipeline run). From there we can gather pretty much every information available from the pipeline (but keep performance in mind ).

I'm not sure though if we can distinguish scheduled pipelines from others (manual, git push, etc.).

@kmann It might be easier to facilitate these discussions in their own issues. You can create an issue for each problem, a, b and c. then link them here. That way the conversation can stay focused on what you are trying to uncover and solve for each independent problem/solution.

Here is a list of open issues that may be useful to see: &275 (closed)

@andyvolpe Since the proposed in this issue is focused on the layout: I included problem A-C together to demonstrate how the current data, as well as data we'd like to add, could live within the proposed layout. I agree we can better facilitate discussions separately, but I don't want to lose the focus on the problem the current layout is causing.

@kmann Ahhh! I got into the weeds! Sorry about that.

no worries, thanks for the suggestion! It makes sense to move problem C out to define what data would help the user.

changed the description

mentioned in issue ux-research#240 (closed)

Following up on the UX group review yesterday. Here are the takeaways and next steps we discussed:

@rverissimo heard you’re working on table styling. Could you link an issue for awareness, so we can align the table in this issue?
Follow up with @jeldergl about any new visual design considerations as needed. Will do when we settle on our direction.
@plafoucriere recommended displaying "new vulnerabilities": https://gitlab.com/gitlab-org/gitlab-ee/issues/12485
Inform our next steps with quick user testing/feedback: ux-research#240 (closed) @tlavi @katokpara
@leipert would like to get FE feedback and constraint considerations. Who would be best to sync with?

@kmann there's no issue for table fixing the table styles yet, as I am still working on the base documentation. Here's the merge request https://gitlab.com/gitlab-org/design.gitlab.com/merge_requests/1444

I'm having troubles with our servers right now and can't access the Security Dashboard. But I would like to know if the component that is currently in use is a table de facto or a div container style as table. Do you happen to know that, Kyle?

@rverissimo looks like the container is: <div> class="ci-table"</div>

marked this issue as related to ux-research#240 (closed)

Hey @NicoleSchwartz @kencjohnston we've been working on this issue, which proposes a layout that considers space for https://gitlab.com/gitlab-org/gitlab-ee/issues/11190 and https://gitlab.com/gitlab-org/gitlab-ee/issues/7521

I propose we work on this in %12.2 and then we have a foundation for the issues above for %12.3. By prioritizing the layout first, these other issues can be implemented faster and without UX and tech debt. Can we plan on this path forward?

Kyle - Andy suggested we move the layout change to EE for prioritization and execution.

@vkarnes great suggestion. We will be doing that shortly, with proposed steps. Related to and following up on the Secure product/UX meeting today with @NicoleSchwartz: we will follow up with FE feedback from @samdbeckham, as we'll be reviewing layout tomorrow.

Perfect, thanks!

These are great, thanks @kmann . I'd like to see this redesign make progress. Are there specific issues we can schedule for upcoming releases that would step us towards this design? cc - @leipert

@kencjohnston as far as I know the next step for this is user-testing @kmann's two design options against each other as well as against the existing dashboard design. These tests will be facilitated by both myself and Kyle on the week of the 22nd. We will test with 8 participants: 4 security analysts and 4 developers. See the issue for that study here.

@kencjohnston thanks. Next steps: working with @samdbeckham to put together a new issue that is UX ready and can be implemented by FE in one milestone. This change supports and helps solves for data display consideration needed for https://gitlab.com/gitlab-org/gitlab-ee/issues/11190 and https://gitlab.com/gitlab-org/gitlab-ee/issues/7521. Will follow up Monday with the new issue and proposal for next steps.

@tlavi @kmann - So to clarify, we aren't going to do user-testing, we'll implement the designs in an MVC and receive feedback from users. That work for everyone? Iteration FTW.

@kencjohnston no problem, I was about to send a screener to find potential participants to test this pre-implementation, but will avoid doing so given your steer.

cc @kmann

MVC proposal: https://gitlab.com/gitlab-org/gitlab-ee/issues/12846

I had a quick play about with the existing template to see how easy it would be to align it to this design.

After about 10 minutes I got 80% of the way there, but the chart will need some work to get it looking good at that smaller size.

There's also a bit of a whitespace issue along the right side of the vulnerability list as the list is a lot longer than the chart and the right side of the list is pretty blank until you start interacting with it.

Gah, I left in the counts.

Here's another:

I'm a bit late to the discussion, but the "Additional considerations" (having sparklines to show changes over time) is a big win in this recommendation. It makes this part lighter and helps the user to focus on what matters there: We care about trends, not about comparing different severities with each other.

Thanks @plafoucriere totally agree. I created an issue to follow up on this: https://gitlab.com/gitlab-org/gitlab-ee/issues/12881

Closing issue - implemenation issues created: https://gitlab.com/gitlab-org/gitlab-ee/issues/12846 and https://gitlab.com/gitlab-org/gitlab-ee/issues/12881

closed

marked this issue as related to #401 (closed)

mentioned in issue #401 (closed)

removed 1 deleted label

changed title from Part 2, Experience Recommendations: Security Dashboard to Experience Recommendations - Secure FY21-Q3 - Security Dashboard

changed title from Experience Recommendations - Secure FY21-Q3 - Security Dashboard to Experience Recommendations - Secure FY20-Q3 - Security Dashboard

changed the description

@kmann I want to follow up on the status of the recommendations you have proposed for the UX Scorecard. Have all of your recommendations been implemented into the product? with the exception of gitlab#12485 (closed)?

@vkarnes Yes, correct gitlab#12485 (closed) was not completed but can be promoted after the completion of 1st-class objects (that was the roadblock).

Completed:

Problem A	Problem B	Problem C

Table view vulnerabilities are pushed down the page below the summary and large chart. The user needs to scroll up and down to adjust filters then view the table data. Example from a 1440x900px display (common laptop)	When the user adjusts the severity filters, for example to "high" only, the summary view still displays “0” for the non-selected criticality levels (chart displays only the filtered object). This occupies space and displays non-relevant data.	It’s unclear what time frame the table vulnerabilities are displaying (here is what it shows: recent security scan on default branch). Also, unclear what projects have or have not been tested and when.

Experience Recommendations - Secure FY20-Q3 - Security Dashboard

Overview

Proposal for consideration and review

Designs

Child items ...

Activity

Experience Recommendations - Secure FY20-Q3 - Security Dashboard

Overview

Proposal for consideration and review

Relates to

Activity