Invalid cloud native build logs

Description

We have successfully rolled out correctness verification mechanisms for build logs. It has proven to work correctly, but unfortunately it detects around one corrupted build log per minute, in gitlab-org/gitlab project.

invalid_chunks_prometheus

Because we now log detected corrupted build logs to Kibana, we do have more information about when and where it happens.

invalid_build_trace_kibana

Without build correctness verification mechanisms we wouldn't be even able to notice corrupted logs like this., but now it is a priority to understand why it happens - multiple reasons are possible, but we should be able to find a root cause here.

/cc @darbyfrey @marin @ayufan @jyavorska