Skip to content

Log large IN lists

Leonardo da Rosa requested to merge 434581-follow-up-from-limit-the-number-of-values-for-in-clause-in-queries into master

What does this MR do and why?

Logs large IN lists in Kibana

As queries containing large IN lists can cause database degradation, we should log these such occurrences to take an action on each case.

How to set up and validate locally

  1. Delete your log/application_json.log file - easier to read

  2. Open rails console

  3. Add the new analyzer tho the analyzer list

Thread.current[:query_analyzer_enabled_analyzers] = [Gitlab::Database::QueryAnalyzers::LogLargeInLists]
  1. Run a query in the console having a IN clause containing more than 2500 ids. The SQL must have more than 10k chars
# 1 match
Project.with_namespace.with_group.include_project_feature.with_group_saml_provider.with_invited_groups.in_namespace(User.first.available_subgroups_with_custom_project_templates((1..3000).to_a)).not_aimed_for_deletion.non_archived
Project.select(:id).where(id: (1..5000).to_a)

# 2 matches
Project.where(id: (1..5000).to_a, namespace_id: (1..3500).to_a).pluck(:id)

# 3 matches
Project.select(:id).where(id: (1..5000).to_a, namespace_id: (1..3500).to_a, project_namespace_id: (1..2700).to_a)
  1. Check the log/application_json.log file

Related to #434581 (closed)

Edited by Leonardo da Rosa

Merge request reports