Extend artifacts definition to provide `reports:` context
Proposal
Proposal for an extension to our architecture that allows us the maximum flexibility that is gonna be used for Junit
reports.
Syntax
We extend artifacts
syntax with:
artifacts:
paths: [....] => artifacts.zip
junit: junit.xml
-
The
paths:
still generatesartifacts.zip
, -
junit
defines single file artifact, uncompressed.
Implementation
- Rails parses
.gitlab-ci.yml
and transforms the CI.yml representation into set of artifacts commands to be executed by runner, - Rails verifies if Runner provide a feature:
multiple-artifacts
send with feature-set when requesting job, - We send to Runner multiple Artifacts specification, with defined Type (archive/junit/codeclimate) / Format (zip, gzip),
- Runner sends all artifacts based on specification,
- Workhorse if does not receive
format
generate metadata, ifformat=gzip
is specified we do not generate metadata, - Rails accepts multiple artifact types, stores them separately, we include
file_types
, - We provide data presenter run in context of Pipeline which accepts and joins multiple archives,
- Presenter generates json result to Frontend, probably with results cached via Sidekiq/ReactiveCache mechanism,
- We can cache data generated in Redis (easy-way), we can cache in Object Storage (slightly harder),
- Rails serve cached data, if data is not present we fire background sidekiq job, data are persisted for limited amount of time (1-2-3 months),
Questions?
- Do we lazilly load data or we persist parsed data?
- Where we cache data? Redis or Object Storage? (Probably Object Storage is the best, and easy achieveable)
11.1
- GitLab-runner: We extend Runner to support multiple-artifacts, some uncompressed,
- GitLab-workhorse: We extend Workhorse to support not generating metadata,
- GitLab-rails: We provide preliminary structure to send relevant data to Runner.
11.2
- We extend
.gitlab-ci.yml
withjunit:
, - We send separate artifacts specification to runner,
- We create presenter to parse and transform the junit report with API agreed with frontend,
- We implement frontend to use the API we provided.
Expected API for junit (or even any other failures), preliminary draft (to verify what junit.xml sends):
{
"summary": {
"total": 100, "failed": 10
},
"details": [ # includes only failed ones
{"file": "/file.c", "line": 10, description: "failed test"},
...
]
}
Some initial discussion here, https://gitlab.com/gitlab-org/gitlab-ce/issues/35379#note_38923310
Edited by Kamil Trzciński