Collect logs based metrics in kubernetes (potentially use mtail)

When working on: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/8962 I noticed we don't have any mtail based metrics, for example: pubsubbeat_errors_total . We'll need many more logs based metrics as we migrate more and more workloads to kubernetes.

The purpose of this issue is to start collecting logs based metrics in kubernetes (potentially use the same tool that we used in the past, i.e. deploy mtail).