Skip to content

Calculate queueing delay with milliseconds

What does this MR do and why?

Addressing Wait time to pick a job is always an integer value (#427433 - closed)

Schema changes will be applied manually similar to Add soft delete option to ClickHouse events table (!133579 - merged)

Note that, support for milliseconds was added in https://github.com/ClickHouse/ClickHouse/pull/51291 which landed on 23.7 https://clickhouse.com/docs/en/whats-new/changelog#-clickhouse-release-237-2023-07-27

ClickHouse Cloud instances we use run on 23.8

Screenshots or screen recordings

Screenshot_2023-10-10_at_11.34.53

How to set up and validate locally

  1. Update your clickhouse: (for macs just execute this line in the clickhouse directory https://clickhouse.com/docs/en/install#quick-install, it will replace the binary).
  2. https://docs.gitlab.com/ee/development/database/clickhouse/clickhouse_within_gitlab.html
  3. Execute the following query in CH console:
INSERT INTO ci_finished_builds (id, project_id, pipeline_id, status, finished_at, created_at, started_at, queued_at, runner_id, runner_manager_system_xid, runner_run_untagged, runner_type, runner_manager_version, runner_manager_revision, runner_manager_platform, runner_manager_architecture) SELECT
    number AS id,
    toUInt64(1 * pow(1 - (rand() / 4294967295.), -1 / 1.)) AS project_id,
    toUInt64(1 * pow(1 - (rand() / 4294967295.), -1 / 1.)) AS pipeline_id,
    ['success', 'success', 'success', 'success', 'success', 'failed', 'failed', 'cancelled'][1 + (rand() % 8)] AS status,
    toDateTime64(now(), 3) - toIntervalSecond(rand() % (86400 * 30)) AS finished_at,
    (finished_at - toIntervalSecond(-600 * ln(1 - (rand() / 4294967295.)))) - toIntervalMillisecond(-1000 * ln(1 - (rand() / 4294967295.))) AS started_at,
    (started_at - toIntervalSecond(-10 * ln(1 - (rand() / 4294967295.)))) - toIntervalMillisecond(-1000 * ln(1 - (rand() / 4294967295.))) AS queued_at,
    (queued_at - toIntervalSecond(-10 * ln(1 - (rand() / 4294967295.)))) - toIntervalMillisecond(-1000 * ln(1 - (rand() / 4294967295.))) AS created_at,
    rand() % 10000 AS runner_id,
    toString(rand() % 1000) AS runner_manager_system_xid,
    rand() % 2 AS runner_run_untagged,
    (rand() % 3) + 1 AS runner_type,
    '' AS runner_manager_version,
    '' AS runner_manager_revision,
    '' AS runner_manager_platform,
    '' AS runner_manager_architecture
FROM numbers(30000)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vladimir Shushlin

Merge request reports