Backport of 'Fix ci_finished_builds engine swap on deferred post-deploy' into 18.10

Narendranrequested to merge
backport-fix-ci-finished-builds-engine-swap-18-10 into 18-10-stable-ee

What does this MR do and why?

Backports !240219 (merged), !240627 (merged), and !240798 (merged) (merged commit b4f205d5) to 18-10-stable-ee, plus a stable-branch-only schema-order fix.

ClickHouse post-deployment migration 20260108073937_alter_ci_finished_builds_engine_with_version failed in three ways on stable branches that run ClickHouse 23.x/24.x:

  1. Structure mismatch (#593129 (closed)): Code: 122. DB::Exception: Tables have different structure. (INCOMPATIBLE_COLUMNS) when self-managed customers upgrade from 18.8.x to 18.9.x–19.0.x. Root cause: hardcoded CREATE TABLE for the tmp table; later-timestamped migrations add columns/projections before this post-deploy migration runs. Fixed in master by !240219 (merged).

  2. Missing PARTITION BY / ORDER BY on CH 23.x/24.x: Code: 36 BAD_ARGUMENTS because the CREATE TABLE … AS source ENGINE = … form does not implicitly inherit those from the source on older ClickHouse. Fixed in master by !240627 (merged).

  3. drop_tmp_table UNKNOWN_SETTING on CH 23.x: SETTINGS max_table_size_to_drop = 0 is only valid as a query-level setting on CH 24.0+; on CH 23.x it raises Code: 115 UNKNOWN_SETTING. Worse, simply omitting it would fail DROP for any customer with >50 GB in tmp. Fixed in master by !240798 (merged): on CH 23.x DETACH every partition first (size check does not apply), then DROP the empty table.

Additionally, db/click_house/main.sql on this branch has projection order drift (the build_stats_* projection appears before by_project_status_v2 / by_runner_owner_v2, while master and 18-11 have the correct order). This pre-existing drift was introduced by commit 75fabe9945030dd0 (Mar 10, "Add ClickHouse migration for ai_audit_events table") and fixed on master by 23cf5be24ce99c but never backported to 18-10. Without this fix clickhouse:check-schema fails. Cherry-picked here as a stable-branch-only schema fix.

Final commit list (4 cherry-picks):

commit from MR / source purpose
ca503ffb !240219 (merged) Initial fix: clone source structure via CREATE TABLE … AS source
d811b49f !240627 (merged) Explicit PARTITION BY / ORDER BY for CH 23.x/24.x compatibility
b4f205d5 !240798 (merged) DETACH+DROP for CH 23.x, version-agnostic spec, dedup-gate threshold corrected to CH 24.8+, self-healing create_tmp_table, and a .pluck('name') rubocop fix for rubocop-rails 2.26+.
23cf5be2 stable-branch-only Fix projection order in db/click_house/main.sql (master already has it)

Verified end-to-end on CH 23.11.3, 24.9.1 and 25.x via docker — full migration sequence succeeds; row count preserved; engine swapped from ReplacingMergeTree(deleted) to ReplacingMergeTree(version, deleted).

References

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

  • This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
  • The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
  • The MR title is descriptive (e.g. "Backport of 'title of default branch MR'"). This is important, since the title will be copied to the patch blog post.
  • Required labels have been applied to this merge request
  • This MR has been approved by a maintainer (only one approval is required).
  • Ensure the e2e:test-on-omnibus-ee job has succeeded, or if it has failed, investigate the failures. If you determine the failures are unrelated, you may proceed. If you need assistance investigating, request help in the #s_developer_experience Slack channel to confirm the failures are unrelated to the merge request.

Note to the merge request author and maintainer

If you have questions about the patch release process, please:

Edited by Narendran

Merge request reports