Add prometheus gauges for in-flight requests and running "commands"
In https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7390 we're talking about alerting on gitaly open file descriptors. @bjk-gitlab pointed out that it would be better if we could normalize that metric. It seems the best thing to normalize it by is the number of Gitaly client connections. Unfortunately, we don't have a good metric for that: this appears to be hidden inside the grpc-go library.
However, if look at the incident that prompted https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7390, we see that the problem had to do with too many processes. We could do the following to get normalized alerting on that:
- track in-flight grpc requests via a gitaly server middleware, with a prometheus gauge
- track running "commands" in gitaly/internal/command: this is the central abstraction we use to run sub-processes in gitaly. Nothing stops us from adding a gauge there.
Then once we have these two gauges, we could alert on the between those two. (Note that in-flight requests may be zero. The alert should not blow up on that.)