Setup an OTEL collector to send traces to observe.gitlab.com
Summary
We need to setup an otel collector to receive pipeline traces and upload them to GitLab Observability.
Here is the official documentation describing an OpenTelemetry Collector: https://opentelemetry.io/docs/collector/.
Proposal
In #2720 (closed), we will send pipeline event webhooks to delivery-metrics. Delivery metrics will trigger a rake task that traces a pipeline when it receives an event indicating that a deployment pipeline has completed.
The rake task will upload pipeline traces to the OTEL collector implemented by this issue, and the OTEL collector should upload it to GitLab Observability with the appropriate token authentication.
These docs about running an OTEL collector in k8s might be useful: https://gitlab.com/gitlab-org/opstrace/opstrace/-/blob/main/docs/guides/user/sending-traces-with-otelcollector.md#example-kubernetes-deployment.
This comment (#2720 (comment 1196693617)) describes how to run an OTEL collector locally.
-
Create a CI job that has the OTEL collector running as a service within it. The CI job will eventually be used toruns the rake task being added by gitlab-org/release-tools!2110 (merged).
Update
Using services turned to be not a viable solution as we cannot pass in variables to the command
setting to the otel-collector docker image in the service.
We also attempted to spin up docker-in-docker manually using the script
section in the job, however we got into further complications with implementing docker properly inside of the Dockerfile. We were able to install docker binary via Dockerfile, but the daemon was not being properly started.
The above attempts can be seen in the MR in ops https://ops.gitlab.net/gitlab-org/release/tools/-/merge_requests/5/ for future reference.
To not further delay this issue, we decided to do the faster solution. We are no longer aiming to run OTEL collector on the CI, but to run the rake task on the CI to directly send metrics to observe.gitlab.com
.