Supporting datadog's agents for metrics collection
Datadog has a set of open source agents, datadog-agent and dogstatsd, that are responsible for sending metrics from a system up to the Datadog service. It has a massive number of integrations for grabbing system metrics, and is relatively easy to instrument your code with.
Interestingly, Datadog already supports scraping Prometheus instrumented applications: https://docs.datadoghq.com/agent/prometheus/
It would be great however if we could leverage these and get it to send metrics to Prometheus. Let's tackle these metrics agents individually.
datadog-agent
This is the primary metrics generation source for Datadog, and is what powers all of their integrations. It is also possible to instrument your own application with this agent using python checks.
The protocol appears to be undocumented, I was not able to find docs or source code.
Also, there does not seem to be any existing libraries that can do translation to other formats. A very common service for this, telegraf, only supports outputting to datadog. However to do this, it calls the Datadog webhook API rather than support the native protocol.
dogstatsd
dogstatsd is responsible for sending custom application metrics up to datadog, and uses a fork of the statsd protocol. It requires you to instrument your own application.
It has wide language support. For example, in the case of Ruby, you can generate the statsd metrics using dogstatsd-ruby library. This library will generate dogstatsd compatible UDP packets, and try to send them to localhost:8125
, which is where the dogstatsd agent would normally be listening.
Interestingly, the Prometheus statsd-exporter supports the datadog extensions to statsd, and converts them to Prometheus labels: https://github.com/prometheus/statsd_exporter#dogstatsd-extensions.
This means that you can install and use the statsd-exporter in place of dogstatsd, and get Prometheus metrics out of an application that was instrumented for Datadog using their libraries.
It is important to note that this will cause metrics to stop being sent to Datadog, so it will primarily be useful for people looking to migrate from Datadog to Prometheus without having to re-instrument. Or potentially if the Datadog library is higher quality than the Prometheus library.
Path forward
- We can document how to utilize dogstats-d instrumented applications with Prometheus/GitLab, by swapping out dogstatsd for statsd-exporter. This seems like low hanging fruit, but we would want to validate this all works well first.
- We could try to invest time in writing a datadog-exporter for Prometheus, or an input plugin for telegraf.