Document how to add custom metrics in the Self monitoring project

Problem to solve

In #121721 (closed) we are planning to present a small number of metrics in the GitLab Self-Monitoring project and make the project expandable with more metrics as a self-service.

Further details

For this issue we should provide detailed documentation on how we add metrics and we should try to identify any gaps (if exist) in order to make this a self-service process.

Documentation

We should provide clear documentation on how to add new metrics to the dashboard

Success criteria

1. Step by step guide on how to add new metrics
2. List of issues (if exist) for making this process self-service

Links / references

Issue to add the metrics to the dashboard: #121721 (closed)