Datadog CI Visibility integration does not appear to handle manual jobs correctly

Summary

I have a manual "deploy to production" job configured, which we run every two weeks or so. This job was run today, after which our p50 duration for the 'deploy' job jumped up to 6.9 days, even though the runtime of the job itself is more like 5-10 minutes. (This job also runs in other environments, such as dev and staging.)

I think this is because the Gitlab integration is measuring the time between when the pipeline ran (but the job did not, since it requires a manual trigger) and when the job ran, which indeed can be measured in days. Instead, I'd like the measurement to be taken starting from the time the job is triggered.

I've also reported this as an issue to Datadog support; not 100% sure where the problem lies here.

Steps to reproduce

  1. Configure a CI pipeline with a job that runs both automatically and manually. Keep the run time of the job to a few seconds.
  2. Configure the Datadog integration.
  3. Trigger the automatic job (say, via push or merge request) several times. Observe metrics for the job and pipeline time, which should show just a few seconds of run time.
  4. Wait for several minutes.
  5. Trigger the job manually. Observe a sudden jump in the job's run time metrics.

Example Project

I can't show the entire end-to-end repro without sharing access to our Datadog account, but I'll follow up later with a sample pipeline config at least.

TODO

What is the current bug behavior?

The time between when a pipeline runs and when a manual job in that pipeline is triggered is included in the job run time metric sent to Datadog.

What is the expected correct behavior?

Only the actual run time of the manual job should be included in the run time measurement.

Relevant logs and/or screenshots

Job run time spike in Datadog: image

Results of GitLab environment info

Gitlab EE self-hosted, 13.11.2 (with :datadog_ci_integration feature flag enabled)