Coverage visualization not displayed until the cobertura file is opened

changed the description

added automation:ml grouppipeline security labels

This issue was automatically tagged with the label grouptesting by TanukiStan, a machine learning classification model, with a probability of 1.

If this label is incorrect, please tag this issue with the correct group label as well as automation:ml wrong to help TanukiStan learn from its mistakes.

This message was generated automatically. You're welcome to improve it.

Hi @kdemircan , what would be the best way that we could have this issue acknowledged or taken in?.

Hi,

I have the same problem, and I was working on a (public) example project that can be used for reproducing the issue. Here the link to a merge request that is exhibiting the described behavior: https://gitlab.com/klaasdellschaft/inlince-code-coverage/-/merge_requests/1/diffs

When working on reproducing the issue, I made the following additional observations:

When reproducing the issue, it is important that you already display the merge request while the pipeline is still running. Wait on the "Overview" tab of the merge request UI until the pipeline is displayed as finished.
After the pipeline is displayed as finished, click on the "Changes" tab. The changes tab will get loaded (as suggested by a quick loading wheel, a change in the URL, and the network monitor of your browser) and displayed, but without the coverage information.
Assuming that the pipeline is already finished, you may browse around in the Gitlab UI (e.g. for opening the cobertura.xml in the pipeline artifacts) and return to the merge request. Alternatively, you can also hit the "refresh" button of your browser (sometimes I have to hit the refresh button twice). After that, the "Changes" tab will display the coverage information (assuming that no additional pipeline is running in the meantime for the same merge request).

Additional information: It seems to me that "inline" view, as well as the "side-by-side" view are affected by this behavior. It depends on which of the two views you have previously configured as your preference, and thus which one gets displayed first when opening the changes tab.

Overall, given that a simple refresh of the merge request page is finally displaying the coverage information (assuming no pipeline is running for that merge request at that time), I think it is not a hard "bug" but a matter of user experience and user expectation. Given that the Gitlab UI was giving me visual clues that the "Changes" tab gets loaded at the point of first clicking on it, it did not occur to me to hit the refresh button for reloading it again in order to see the coverage information. Further strange behavior is that I have to hit the refresh button twice sometimes (in Firefox as well as in Edge).

Furthermore, given that the UI is automatically refreshing the pipeline status (and finally displaying it as finished), it may also create the expectation in the user that other information that depends on the pipeline status is also getting refreshed. For example, also the test summary report displayed on the overview tab is getting refreshed as soon as the pipeline is displayed as finished. So even if the UI wouldn't have given me visual clues of freshly loading the "Changes" tab, I think it is reasonable to expect that the coverage information is getting automatically refreshed in the background.

Hi @u809693

could you maybe add the link to my example project / merge request to the original issue description? Then a developer / tester does not have to read all comments in order to discover that link. Maybe you could also add the additional details to the original issue description.

@klaasdellschaft @u809693 - Hi thanks for the issue! When I review the MR I see the inline coverage displaying on the line shown in the screenshot without opening the cobertura.xml file. Is this happening every time you load the MR or only after a pipeline runs?

The processing of the file happens after upload so we don't delay load of the Changes tab / the MR page so it may take some time for a file to be processed and the visualization to appear.

I think this is working as expected with the delay being related to processing. Better documentation around this is needed it seems. I'm going to close the issue out and I've opened an MR to add to the docs.

Hi @jheimbuck_gl

thanks for looking at the issue. However, I'm not so sure whether this problem is really related to a processing delay on the server side. Could you maybe try once again to reproduce it?

Just looking at the existing merge request is not sufficient. In order to experience the behavior, (1) you have to start a new pipeline that recomputes the coverage report, and (2) you have to initially load the merge request in your browser while the pipeline is still running. When looking at a merge request where the pipeline last ran 6 days ago, you will not experience the described behavior. Overall, given my experimentation after this issue had already been created, I would say that the title "... until the cobertura file is opened" is not capturing / describing the problem properly.

I would suggest the following steps, with which I have a 100% success ratio in reproducing the behavior, and excluding the possibility of it being caused / explained by a processing delay:

Open https://gitlab.com/klaasdellschaft/inlince-code-coverage/-/merge_requests/1 in two different browsers. Do not go on the "Changes" tab of the MR yet.
Start a new pipeline by pushing some change to the MR. I added you as a project member, so feel free to edit any file in order to start a MR pipeline.
Use one of your browsers to (1) confirm that the pipeline is finished, that (2) the report has been processed, and (3) that the coverage information gets displayed.
Change to your other browser. Click on the "Changes" tab of the MR. A loading wheel as well as the network analysis tool of your browser will show you that the data of the changes tab gets loaded now for the first time (i.e. it should be "fresh" and thus the coverage report should be available). However, the coverage report is not getting displayed despite being already available in the other browser.
In your browser that does not display the coverage information yet, hit the "refresh" button of your browser. In almost all cases (i.e. I would estimate in 80%-90% of cases), the coverage information will still not be displayed, even when waiting several seconds for any asynchronous Javascript / rendering to finish. After hitting the "refresh" button the second time, finally the code coverage will get displayed.

On my machine, I'm using Firefox and Edge as browsers during the experiment. But @u809693 has mentioned Chrome in the original issue description, so I think the problem should also appear in Chrome.

@klaasdellschaft - I just ran that test on https://gitlab.com/klaasdellschaft/inlince-code-coverage/-/merge_requests/2/diffs and after seeing the pipeline finish and new coverage information appear on in Chrome I logged into my account on Safari, went to the MR and the changes tab and the coverage information is displayed.

Hi @jheimbuck_gl

because you have problems reproducing the behavior on your side, I did a screen recording that demonstrates the behavior. Both recordings are done on a Windows 10 machine with Firefox browser.

Scenario 1: Visit the merge request minutes after the pipeline has finished

The pipeline has finished 3 minutes ago.
Navigate to the changes tab of the MR view
After loading, wait a few seconds for delayed rendering => nothing happens
Hit the refresh button => now the coverage report immediately appears

Scenario 2: Wait in MR view for the pipeline to finish

Open the MR view while the pipeline is still running
Wait for pipeline to be finished + "Test summary" widget being updated based on finished pipeline
Navigate to the changes tab => No coverage report is rendered
Hit the refresh button the first time => Still no coverage report
Hit the refresh button a second time => Coverage report appear immediately
Additional info: In the recording, I'm proceeding pretty fast to the changes tab so that I keep the recording short. However, a double refresh for the coverage to appear is even required when I'm waiting for example 30 seconds or more between test results being rendered and going to the changes tab.

Thank you for the detailed write up @klaasdellschaft I think this is the report still being processed as it is not guaranteed to happen during pipeline runtime. @mfluharty how often are we polling on the changes tab for the report data and is there a way to display that without refresh maybe?

Hi @jheimbuck_gl

thanks for your follow-up. What would be your expectation regarding a typical background processing delay? Are we talking about up to a minute? Or several minutes? Is the background processing started immediately after the pipeline having finished, or upon first user navigating to the Changes tab?

I think two observations in my scenarios make the background job delay explanation less likely:

In scenario 1, I'm navigating to the MR 3 minutes after the pipeline has finished. Still, the coverage is not getting displayed on first view. Instead, I have to do a refresh to see it. In contrast, in scenario 2 I can see the coverage roughly 30 seconds after the pipeline has finished, as long as I remember to do a double refresh.
In scenario 2, the screen recording is not waiting long until I do the double refresh. However, in a fresh run, even if I wait longer (e.g. roughly 3 minutes after pipeline having finished), I still need to do a double refresh in order to see the coverage report being displayed.

Overall, given the example MR and its size of the diff / source code / coverage report, I would assume that the background job finishes within seconds after the pipeline (in scenario 2, I was able to see the coverage 30 seconds after pipeline having finished; I just had to do a double refresh). However, even if I wait minutes after the pipeline having finished before navigating to the MR and/or the changes tab, the coverage report never appeared "spontaneously". I always had to do 1-2 refreshes.

@klaasdellschaft I'm not sure the page auto refreshes with the coverage information, this may be the disconnect i'm having. I'll sync with the team and let you know what I find.

changed the description

Setting label(s) devopsverify sectionops based on grouptesting.

added devopsverify sectionops labels

mentioned in issue gitlab-org/quality/triage-reports#4593 (closed)

mentioned in merge request !70130 (merged)

closed

Coverage visualization not displayed until the cobertura file is opened

Summary

Steps to reproduce

Example Project

What is the current bug behavior?

What is the expected correct behavior?

Relevant logs and/or screenshots

Results of GitLab environment info

Designs

Child items ...

Activity