GitLab Jaeger integration experience

Description

I like GitLab Jaeger integration. Distrubuted tracing system allows us to easyly spot slow endpoints in all microservices.

However, I was struggling to debug a slow endpoint today, and I wonder if it's just because I didn't know how to use Jaeger or it's not supported yet.

Experience

Today, I ran a GitLab instance on my local environment and found a slow endpoint.

Screenshot_from_2019-04-16_09-33-47

and I found a corresponding trace in Jaeger:

Screenshot_from_2019-04-16_09-28-39

Took a look closely. It seems rendering a template took about 2 sec. However, I had no idea about the details of the main cause of the slowness (See the screenshot below). Is there any best practice for the further investigation? Off the top of my head, this could be due to slow database query.

Screenshot_from_2019-04-16_09-29-56

Screenshot_from_2019-04-16_09-30-44

Also, there are empty space for the last 1.4 secs. Did workhorse wait for the other services which doesn't support Jaeger yet? I had no clue.

Screenshot_from_2019-04-16_09-31-05