Backport of 'Disable async_insert in build and pipeline sync operations'

What does this MR do and why?

Backport of !219352 (merged) to 18-7-stable-ee.

Disables async_insert in ClickHouse build and pipeline sync operations to prevent data duplication in materialized views.

Problem: The ci_finished_pipelines_daily materialized view shows significant data inflation (up to 80% for some months) compared to the source ci_finished_pipelines table. Investigation revealed that HTTP read timeouts during async inserts cause the client to believe insertions failed, triggering retries that result in duplicate data.

Root cause: The async insert settings (async_insert=1, wait_for_async_insert=1) combined with ClickHouse's wait_for_async_insert_timeout=120s can exceed the Ruby HTTP client's read timeout. When this happens:

  1. The HTTP client times out waiting for a response
  2. The insertion may have actually succeeded in ClickHouse
  3. Sidekiq retries the job, inserting the same data again
  4. The source table (ReplacingMergeTree) deduplicates, but the MV (AggregatingMergeTree) accumulates duplicates

Solution: Remove async_insert settings entirely. Since we already batch data before insertion, async insert provides minimal benefit while introducing timeout-related reliability issues.

References

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

  • This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
  • The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
  • The MR title is descriptive (e.g. "Backport of 'title of default branch MR'"). This is important, since the title will be copied to the patch blog post.
  • Required labels have been applied to this merge request
  • This MR has been approved by a maintainer (only one approval is required).
  • Ensure the e2e:test-on-omnibus-ee job has succeeded, or if it has failed, investigate the failures. If you determine the failures are unrelated, you may proceed. If you need assistance investigating, reach out to a Software Engineer in Test in #s_developer_experience.

Note to the merge request author and maintainer

If you have questions about the patch release process, please:

Edited by Pedro Pombeiro

Merge request reports

Loading