Export CI job logs to Datadog
Description
As part of the work to implement GitLab’s Datadog Integration, the Datadog team is looking to import all logs from CI jobs into Datadog.
The Datadog team came up with several implementation alternatives. The option that looks the most feasible is to extend the internals of incremental logging:
This idea would be to extend the internals of the Incremental Logging Architecture, so that when pieces of logs are written to redis, they are also exported to Datadog. They would be sent on direct requests to Datadog’s intake endpoint for logs. The requests could use the same API keys we have configured for webhooks.
Pros: Low implementation effort on Datadog. Incremental logging would also be available within Datadog.
Cons: High implementation effort on GitLab. It might involve some configuration on GitLab
Possible modifications: Depending on the solution design of this, it could be a new feature of GitLab for more advanced setups on logs. Writes could go through a backend agnostic collector like fluentd, or through syslog.
Release Notes
GitLab's DataDog integration will now support a new hook type sent to Datadog through a new checkbox in the integration settings. This new hook type sends a pre-signed link to download the job trace log from object storage.