JUnit export documentation
There are two pages related to JUnit test export files as job artifacts, which show up as MR widgets: https://docs.gitlab.com/ce/ci/yaml/README.html#artifactsreportsjunit and https://docs.gitlab.com/ce/ci/junit_test_reports.html. I'd like to suggest a couple of improvements:
-
On https://docs.gitlab.com/ce/ci/yaml/README.html#artifactsreportsjunit there is a link to https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html. This link does not appear on https://docs.gitlab.com/ce/ci/junit_test_reports.html, which is otherwise more extensive. I'd expect the latter doc page to include all information from the former, and more. -
The link pointed to is somewhat confusing, however. I was implementing JUnit export for a testing framework which did not have this export function yet, and that link led me to believe that all the attributes mentioned there were relevant to GitLab somehow. This is not the case at all. For instance, the filename and line number are not used (while you could imagine that the "failed test" message would then link to the offending file). In the example at the bottom of the IBM page it looks like the contents of <failure>
are key-value pairs separated by a colon, while in fact GitLab allows you to put anything there (and does not parse such key-value pairs). (So I spent some time getting my system to spit out filename and line number, which got quite hairy, and turned out to be pointless😀 .) I would suggest that the GitLab docs be extended with an explanation about what attributes are considered exactly. -
This ties into the fact that some fields in these report files are handled specially. For example, there is a classname
attribute which will somehow be shown (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28376), but it is not clear to me what it is meant for (I cannot find it in the docs). There are some files liketest_case_entity.rb
,test_case.json
, andstate.js
, which seem to contain lists of different keys, but how they are handled does not become clear. The Q&A on https://stackoverflow.com/q/442556 contains some links which are more complete than the IBM spec, perhaps they are helpful. -
The text under "overview" and "use cases" on https://docs.gitlab.com/ce/ci/junit_test_reports.html say roughly the same thing. Perhaps they can be merged. -
The "use case" is also not really a use case, it describes the transition from without JUnit export (in 2) to with JUnit export (in 4). -
https://docs.gitlab.com/ce/ci/junit_test_reports.html#how-to-set-it-up refers to https://en.wikipedia.org/wiki/JUnit#Ports for a list of languages "on JUnit tests". Not all test frameworks in that list support exporting in this XML format (e.g. Haskell's HUnit). The link is useful, but perhaps it should be introduced differently - as it stands now it seems all items in the list can export this XML format. -
The "limitations" section on https://docs.gitlab.com/ce/ci/junit_test_reports.html#limitations refers to an issue that has now been closed, I think it can be removed.
I realize somebody implementing export functionality in a new test framework is not a typical user story, so I understand if the documentation will not go in depth. An example of a 'GitLab-optimized' export file would be very nice however.
Finally, please note: this feature is great!
Related
https://gitlab.com/gitlab-org/gitlab-ce/issues/17081
Who can address the issue
@dosuken123 You worked on https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21003 which introduced this, could you have a look? Otherwise @fabiopitino, you added https://gitlab.com/gitlab-org/gitlab-ce/commit/24a81b3dccef87cf4c6837a26fadd702b0418d72?