Backport of 'Disable async_insert in build and pipeline sync operations'
What does this MR do and why?
Backport of !219352 (merged) to 18-7-stable-ee.
Disables async_insert in ClickHouse build and pipeline sync operations to prevent data duplication in materialized views.
Problem: The ci_finished_pipelines_daily materialized view shows significant data inflation (up to 80% for some months) compared to the source ci_finished_pipelines table. Investigation revealed that HTTP read timeouts during async inserts cause the client to believe insertions failed, triggering retries that result in duplicate data.
Root cause: The async insert settings (async_insert=1, wait_for_async_insert=1) combined with ClickHouse's wait_for_async_insert_timeout=120s can exceed the Ruby HTTP client's read timeout. When this happens:
- The HTTP client times out waiting for a response
- The insertion may have actually succeeded in ClickHouse
- Sidekiq retries the job, inserting the same data again
- The source table (
ReplacingMergeTree) deduplicates, but the MV (AggregatingMergeTree) accumulates duplicates
Solution: Remove async_insert settings entirely. Since we already batch data before insertion, async insert provides minimal benefit while introducing timeout-related reliability issues.
References
- Original discussion: https://gitlab.com/gitlab-org/gitlab/-/work_items/586319#note_3013011851
- Parent issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/586319
- Parent MR: !219352 (merged)
- Related MR (disable retries): !219149 (merged)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
- This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
- The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
- The MR title is descriptive (e.g. "Backport of 'title of default branch MR'"). This is important, since the title will be copied to the patch blog post.
-
Required labels have been applied to this merge request
- severity label and bug subtype labels (if applicable)
- If this MR fixes a bug that affects customers, the customer label has been applied.
- This MR has been approved by a maintainer (only one approval is required).
-
Ensure the
e2e:test-on-omnibus-eejob has succeeded, or if it has failed, investigate the failures. If you determine the failures are unrelated, you may proceed. If you need assistance investigating, reach out to a Software Engineer in Test in #s_developer_experience.
Note to the merge request author and maintainer
If you have questions about the patch release process, please:
- Refer to the patch release runbook for engineers and maintainers for guidance.
- Ask questions on the
#releasesSlack channel (internal only). - Once the backport has been merged, the commit changes will be automatically deployed to a release environment that can be used for manual validation. See after merging runbook for details.