Extend artifacts definition to provide `reports:` context

Proposal

Proposal for an extension to our architecture that allows us the maximum flexibility that is gonna be used for Junit reports.

Syntax

We extend artifacts syntax with:

artifacts:
  paths: [....] => artifacts.zip
  junit: junit.xml

1. The paths: still generates artifacts.zip,

2. junit defines single file artifact, uncompressed.

Implementation

1. Rails parses .gitlab-ci.yml and transforms the CI.yml representation into set of artifacts commands to be executed by runner,
2. Rails verifies if Runner provide a feature: multiple-artifacts send with feature-set when requesting job,
3. We send to Runner multiple Artifacts specification, with defined Type (archive/junit/codeclimate) / Format (zip, gzip),
4. Runner sends all artifacts based on specification,
5. Workhorse if does not receive format generate metadata, if format=gzip is specified we do not generate metadata,
6. Rails accepts multiple artifact types, stores them separately, we include file_types,
7. We provide data presenter run in context of Pipeline which accepts and joins multiple archives,
8. Presenter generates json result to Frontend, probably with results cached via Sidekiq/ReactiveCache mechanism,
9. We can cache data generated in Redis (easy-way), we can cache in Object Storage (slightly harder),
10. Rails serve cached data, if data is not present we fire background sidekiq job, data are persisted for limited amount of time (1-2-3 months),

Questions?

1. Do we lazilly load data or we persist parsed data?
2. Where we cache data? Redis or Object Storage? (Probably Object Storage is the best, and easy achieveable)

11.1

1. GitLab-runner: We extend Runner to support multiple-artifacts, some uncompressed,
2. GitLab-workhorse: We extend Workhorse to support not generating metadata,
3. GitLab-rails: We provide preliminary structure to send relevant data to Runner.

11.2

1. We extend .gitlab-ci.yml with junit:,
2. We send separate artifacts specification to runner,
3. We create presenter to parse and transform the junit report with API agreed with frontend,
4. We implement frontend to use the API we provided.

Expected API for junit (or even any other failures), preliminary draft (to verify what junit.xml sends):

{
  "summary": {
     "total": 100, "failed": 10
  },
  "details": [ # includes only failed ones
     {"file": "/file.c", "line": 10, description: "failed test"},
     ...  
  ]
}

---

Some initial discussion here, https://gitlab.com/gitlab-org/gitlab-ce/issues/35379#note_38923310