Validate correctness of traces and log invalid ones

Description

In gitlab-runner#26545 (closed) we described a new mechanism of verifying correctness of CI/CD build traces sent by a runner to gitlab-rails, in an incremental way.

We are trying to make Live Traces production ready and generally available to unblock migration to kubernetes &3791 (closed)

Proposal

Upon build completion, a runner will send data with CRC32 of a trace.

Using "CRC32 combine" algorithm retrieve CRC32 checksums of trace chunks and check if the checksum matches. In case of a mismatch increment a corresponding metric using a Prometheus counter.

/cc @darbyfrey @cheryl.li

Edited by Grzegorz Bizon