Distributed Tracing in LabKit
-
Review changes -
-
Download -
Patches
-
Plain diff
Reviewers Guide
This change adds distributed tracing to LabKit. LabKit is embedded within Gitaly and Workhorse (and could potentially be used with other Go services at GitLab too).
Design Decisions
Optional OpenTracing Support
The Opentracing API doesn't ship with the ability to switch tracing vendors. It does not support loading implementation libraries dynamically, in the way you can with JDBC, ODBC, etc.
From the application developer's point-of-view, this has several disadvantages:
- Most GitLab users do not need distributed tracing on their GitLab instance.
- Therefore, adding Jaeger (+ thrift and other downstream dependencies) to all Workhorse + Gitaly installs is not ideal.
- Other than GitLab.com, any other major GitLab instances may already have their own distributed tracing solutions (Lightstep, Zipkin, Datadog, etc) which they have invested in and have experience with. Forcing them to use the one of our choice is not ideal.
- We may use different tracing libraries in Production and Development
For these reasons, I would prefer to ship Workhorse and Gitaly without tracing by default, but allow it to be added when necessary.
My initial attempt was to use the pkg/plugin
go standard package to dynamically load the tracing library when necessary. However, it turns out that this approach is far too fragile. Go plugins would result in lost productivity and support overhead.
Instead, I've gone with conditional build tags so that it's possible to statically link one or more implementation into the binary (Jaeger, Datadog, Lightstep, Zipkin, etc).
To statically compile against a specific implementation, the following approach can be taken:
go build -tags "tracer_static tracer_static_jaeger" ./...
go build -tags "tracer_static tracer_static_datadog" ./...
go build -tags "tracer_static tracer_static_ligthstep" ./...
- etc
Running the examples
The change ships with several examples:
-
example/run-jaeger-static
- start jaeger, and run demo app with jaeger statically linked in -
example/run-datadog-static
- start dd-agent, and run demo app with datadog statically linked in -
example/run-no-tracing
- exclude opentracing altogether
In each case the same application will execute. It's a tiny app running on port 8080. It has a single endpoint:
$ curl http://localhost:8080/query?ttl=100
Hello
All the service does is call itself recursively (via HTTP) as many times as given by the ttl
parameter. To recursively call the application 10 times, use ttl=10
, etc.
After issuing some requests using curl
, it's time to take a look at the traces!
For Jaeger, open http://localhost:16686/search/ and find a trace. Hopefully you should find something like this:
For Datadog, log into https://app.datadoghq.com/apm/home, where you should get something like
OpenTracing "Connection Strings"
In order for this change to be adopted, it needs to be easy to use. For this reason, one of the early decisions I made is that all GitLab components, Ruby and Golang should share a single configuration for their opentracing config.
Unfortunately, there is no common approach to configuring an opentracing implementation.
I've elected to go with a "connection string" approach. Since the same connection string can be used to configure Gitaly (golang), Workhorse (golang), Sidekiq (ruby) and Rails (ruby), it's easy to enable when running in GDK: simple run GDK with GITLAB_TRACING=opentracing://jaeger gdk run
.
All the child processes will inherit the same GITLAB_TRACING
configuration.
Here are some examples of connection strings:
opentracing://jaeger
opentracing://jaeger?debug=true
opentracing://jaeger?sampler=const&sampler_param=1
opentracing://jaeger?sampler=probabilistic&sampler_param=0.1
opentracing://jaeger?http_endpoint=http%3A%2F%2Flocalhost%3A14268%2Fapi%2Ftraces
opentracing://jaeger?udp_endpoint=10.0.0.1:1234
opentracing://lightstep?access_token=123
opentracing://datadog
Note that in order to allow forwards compatibility, opentracing provider factories will ignore unrecognised parameters, unless the strictConnectionParsing=1
option is passed in.
File Sizes
- App with jaeger statically linked: binary: 9604k
- App with datadog statically linked: binary: 8880k
- No tracing: binary: 8124k
Downstream Merge Requests
- Workhorse gitlab-workhorse!325 (merged)
- Gitaly gitaly!976 (merged)
- GitLab-CE: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21280
Merge request reports
- version 409c613f6d
- version 39705250dd
- version 3890361569
- version 3775cb8eb5
- version 360e2cab39
- version 350e2cab39
- version 3424666e3a
- version 33e0f916bb
- version 32a6d91811
- version 312df04c24
- version 304ff354e9
- version 29a9f01255
- version 2867ca6d7b
- version 27ed89b7f8
- version 2623f2d89d
- version 250080fdc3
- version 243a481e4f
- version 23c1b3275b
- version 2229625136
- version 21d1d36b37
- version 207d783e77
- version 19f8369045
- version 182c4bf4f9
- version 1787ab5e36
- version 160e5900fb
- version 15434c2302
- version 14166873ee
- version 130c1fff30
- version 129fa92a16
- version 111e154004
- version 101a8140bb
- version 97c198aef
- version 8ea5ad089
- version 7a85909eb
- version 6f802fc3d
- version 5a678ee75
- version 43091b6f1
- version 3dfd7df4e
- version 2dfd7df4e
- version 11b27ecb6
- master (base)
- latest versiond695bc5a38 commits,
- version 409c613f6d37 commits,
- version 39705250dd36 commits,
- version 389036156935 commits,
- version 3775cb8eb534 commits,
- version 360e2cab3933 commits,
- version 350e2cab3934 commits,
- version 3424666e3a33 commits,
- version 33e0f916bb31 commits,
- version 32a6d9181130 commits,
- version 312df04c2430 commits,
- version 304ff354e929 commits,
- version 29a9f0125528 commits,
- version 2867ca6d7b28 commits,
- version 27ed89b7f827 commits,
- version 2623f2d89d26 commits,
- version 250080fdc325 commits,
- version 243a481e4f24 commits,
- version 23c1b3275b24 commits,
- version 222962513624 commits,
- version 21d1d36b3723 commits,
- version 207d783e7723 commits,
- version 19f836904522 commits,
- version 182c4bf4f921 commits,
- version 1787ab5e3620 commits,
- version 160e5900fb19 commits,
- version 15434c230218 commits,
- version 14166873ee17 commits,
- version 130c1fff3016 commits,
- version 129fa92a1615 commits,
- version 111e15400414 commits,
- version 101a8140bb13 commits,
- version 97c198aef12 commits,
- version 8ea5ad08911 commits,
- version 7a85909eb10 commits,
- version 6f802fc3d8 commits,
- version 5a678ee757 commits,
- version 43091b6f16 commits,
- version 3dfd7df4e3 commits,
- version 2dfd7df4e11 commits,
- version 11b27ecb610 commits,
- Side-by-side
- Inline