Cloud Native Build Logs resiliency improvements
## Description
In https://gitlab.com/groups/gitlab-org/-/epics/4275 we have concluded Cloud Native Build Logs rollout on Gitlab.com.
At this point we consider this feature to be production ready, but we still would like to improve it to make more resilient, especially to a significant replication lag and Redis failovers.
This epic tracks the work that needs to be done to make this feature even more resilient and fault tolerant.
## Description
This epic tracks work that needs to be done to enable Cloud Native Build Logs (Live Traces) on gitlab.com to remove one of NFS dependencies and unblock Kubernetes migration.
For more information about what Cloud Native Build Logs are and how the new architecture looks like, please see architecture blueprint (draft merge request) - https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/59964
Epic about making Cloud Native Build Logs generally available, that tracks an extension of this work, is available here :arrow_right: https://gitlab.com/groups/gitlab-org/-/epics/3791
## Roadmap
<table>
<thead>
<tr>
<th>Work item</th>
<th>Issue</th>
<th>Status</th>
<th>DRI</th>
</tr>
</thead>
<tbody>
<tr>
<td>Observability improvements post-rollout</td>
<td>https://gitlab.com/gitlab-org/gitlab/-/issues/273444</td>
<td>:heavy_check_mark: done</td>
<td>@grzesiek</td>
</tr>
<tr>
<td colspan=2>
–› Send build trace bytesize in the final build update
</td>
<td>:heavy_check_mark: [done](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2521)</td>
<td>@grzesiek</td>
</tr>
<tr>
<td colspan=2>
–› Detect corrupted build logs and increment relevant metric
</td>
<td>:heavy_check_mark: [done](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49004)</td>
<td>@grzesiek</td>
</tr>
<tr>
<td>Replication lag resiliency</td>
<td>https://gitlab.com/gitlab-org/gitlab/-/issues/271253</td>
<td>:footprints: In progress</td>
<td>@grzesiek</td>
</tr>
</tbody>
</table>
epic