Sasha and the team have worked to create browser tests using Selenium and Webdriver.io. Today those tests capture a screenshot as an artifact at failure to help troubleshoot the test issue. Sasha has to dig through the logs and screenshots stored as artifacts to find the failures which takes too long and is done outside of the GitLab UI.
N/A for the MVC the file will be a link to the file.
Permissions and Security
editing the gitlab-ci.yml should follow project guidelines, this requires no special permissions
viewing and downloading the data requires no special permissions
Documentation
This will need a new page in the docs to instruct users:
How to create junit xml that is detected and how to setup artifacts that are discoverable by the feature
Where to see the report
How long the artifacts persist
Common troubleshooting tips if the report fails (like if you need two MRs before the report appears, etc.)
Testing
test for performance with many (1000's) of screenshots
test for junit is present but screenshot is missing
What does success look like, and how can we measure that?
We will measure success by tracking views of the screenshots tab. Since this does not yet exist we will establish a baseline at 30 days after launch and iterate towards improvements.
What is the type of buyer?
The buyer for this is a manager who is trying to make life easier for a team who is doing some automated browser testing and having issues with identifying why those tests are failing.
@samdbeckham Designs are done in #6061 yes. In this iteration not all of the meta data from the designs will be shown, only the test and link to download the screenshot. I've already broken the tracking out to another issue so i'll remove that from the proposal.
The work for the endpoint was done in #202114 (closed) which it looks like @morefice did and can probably assist on.
Create a app/assets/javascripts/pipelines/pipeline_details_screenshots.js file and call the newly created default function in the app/assets/javascripts/pipelines/pipeline_details_bundle.js file. This will be used to instantiate our new Vue app.
Merge Request 2
To stay consistent with the rest of the pipeline page and to be up to date with current frontend standards, we'll use Apollo for this. We'll need to set up a client resolver to get the list of test failures with screenshots from the REST endpoint.
Create a folder app/assets/javascripts/pipelines/components/screenshots with a screenshots.vue file which will be the parent component.
In the screenshots.vue file, we'll need to fetch the screenshots list with a @client GraphQL query.
Merge Request 3
We can follow the pattern used in the Test Report tab to create the table. The table will have two columns, Name and Screenshot.
We could combine 2 and 3 together, but I figure it's best to play it safe and make smaller MRs.
Given the above, I think the current weight of 3 is reasonable. Similar work has been done recently so it should be straightforward.
Depending how/when we want to fetch this data we should ask our frontend folks to determine if we should include those screenshot in our new summary endpoint. If we need to, then the backendis not ready otherwise we could start to iterate by fetching our old endpoint first. Then work on including our screenshots within our summary.
@morefice Given that the old endpoint is quite slow, do you think it makes sense to use it? Should we come up with a bespoke endpoint for returning all of the tests with associated screenshots instead?
can we snag status as well for that column making it 3 not 2? I'll hold off on updating the proposal above until I hear back.
Yes, that is easy enough to add on.
@rickywiens@morefice I'm not sure if we are able to use the old endpoint. The old endpoint returns the suites, and then when you click on the suites it makes another request so you can see all the test cases for that suite. Based on the designs for this feature it looks like we just have one view of all of the failed tests regardless of which suite they are in. WDYT?
Given that the old endpoint is quite slow, do you think it makes sense to use it?
Oh wait a minute looks like I've spoke to fast. Our old endpoint is not actually that bad considering we are returning only the failing tests. So it's still faster than what we used to have. I think we could use it as a starting point to quick off an async HTTP call once a user visit the pipeline page to prepopulate the data.
Based on the designs for this feature it looks like we just have one view of all of the failed tests regardless of which suite they are in. WDYT?
I should have looked a the mocks
The old endpoint returns the suites, and then when you click on the suites it makes another request so you can see all the test cases for that suite.
I think we are talking about 2 different things here. I think what you are mentioning is our new endpoint the summary we created few months ago.
This endpoint already returns all the suites containing failed test cases with attached screenshots.
@morefice Oh I was confused about which endpoint you were talking about. The endpoint you've now linked to looks good to me! Thanks for the help and clarification.
@shampton - Based on our discussion in the design issue and the new MVC approach of adding the link to the file in the modal can you make any necessary updates to the technical proposal in the issue?
I've scheduled this for %14.0 but with the simpler approach it could be a good candidate to pull forward.
@jheimbuck_gl Since our MVC is moving from a separate tab to the same Tests tab on the pipeline page, we may have to revisit the backend for this. Before we were assuming we could make a request for the whole report (just the failures) and pass in the scope=with_attachment param. We'll now have to get the attachment link in the full test report.
For clarification, we are choosing to show a link to the screenshot in the modal for a test case. Here's a screenshot of where I'm referring to:
This is actually a good news! The backend is in a state where the attachment_url attribute is part of our TestCaseEntity.
This means the frontend is able to fetch the attachment url as part of a test case object if the attachment exists. We just need to display the link of the screenshot which is already exposed 🥳
@jyavorska - The team is getting back to showing screenshots from failed tests. This is the MVC for that, a new tab with the failed tests and link to the screenshot if available.
@jreporter - MVC for bringing screenshots captured at automated test failure into the Pipeline Tab. The full epic has the overall vision for where this goes.
@jheimbuck_gl I am loving how this issue is technically broken up and the scope seems reasonably sized. Well done! This is a Core feature and rationale for that?
Lastly, have you considered using the designs functionality in GitLab? I find it really helpful to show the portions of larger designs that will be rendered in this MVC.
This is core because the buyer is really the IC. An argument could be made that team leads care about the coordination between testers and developers and both seeing the results of a failed test in the pipeline view makes this GitLab Premium but the flip side of that is this is a long existing capability that competitors have (jenkins plugin example) that we should provide.
re: designs. Yes, the referenced item for the full implementation uses the design tab and we are starting to make more use of that in issues. We're still sorting out how to reference portions of a design that get built in the broken down issues which i'm sure @rayana will be able to help us with.
I have verified that this change is working in production.
JUnit report attachments are currently behind the junit_pipeline_screenshots_view feature flag which was put in place last year. I've enabled that feature flag globally on .com. I'm also creating an MR to remove the feature flag entirely, but I'm going to ping a few people to make sure that it's safe to remove. I'm also going to wait a few days before merging it while we watch production to make sure nothing goes wrong with it globally enabled.
The screenshot will be available as a link in the details view of a failed test case, but will not actually be rendered inline or even on the proposed screenshot tab. What you get is exactly that blue link in the screenshot in comment #216979 (comment 568314887).
From reading the documentation I also assumed that it would be rendered inline or somehow displayed without needing to click through one more time.
@shampton Do you think that the wording in the documentation could be improved to reduce the misunderstanding that the attached screenshots are easily visible inline?
@shampton you are correct, the text is as you say. Do you think we could use the example screenshot you posted above in the docs? I think for people - who didn't read every single sentence, like me (I'm sorry) - this might help as a visual representation to what difference it makes.