Supporting datadog's agents for tracing collection
Datadog has a set of open source tracing libraries which are responsible for sending traces up to the Datadog service.
Some of these tracers support auto-instrumentation as well, which makes getting started easy and is a feature not readily available in other open source tracers.
For existing users of Datadog who may be looking to migrate to Jaeger, or for those who may be interested in their auto-instrumentation, it would be great if we could leverage these.
A word on Opentracing & Datadog
Opentracing is architected to abstract the backend specific logic from the language library. For example, you can instrument your code with the opentracing-go library, and snap in any of the opentracing compatible "tracers". For example supported tracers include Datadog, Jaeger, Lightstep, and more.
This in theory allows everyone to contribute to high quality language libraries, and allow easy portability between backends by swapping out the underlying tracing library with just a handful of initialization lines, rather than every place a trace is generated.
While Datadog supports being used in conjunction with an opentracing library, they are also building their own language libraries with exclusive proprietary features. For example the auto-instrumentation is a feature that has only been added to their language library, rather than being committed upstream to the opentracing language library. (Note others are doing this as well, like Elastic.)
Datadog Trace API
Datadog has documented their Trace API, which utilizes a seemingly simple albeit proprietary HTTP API: https://docs.datadoghq.com/api/?lang=python#send-traces
It is worth noting that Jaeger and Zipkin also have their own formats, although Jaeger offers support for Zipkin as well.
Datadog Tracing & Metrics
The Datadog trace library appears to collect the 3 classic tracing metrics: throughput, errors, and latency.
Any other application metric you would like to collect, is done via the classic dogstatsd (https://gitlab.com/gitlab-org/gitlab-ce/issues/51876) agent.
Potential paths forward
- We could contribute a PR to Jaeger to accept the Datadog tracing format, similar to their support for Zipkin. Note that no one has opened an issue for this yet, so demand does not seem high.
- Alternatively write an agent, which would receive the traces similar to the DD agent, but translate them to Jaeger and forward them to a Jaeger agent.