Recover gracefully when issuable counts are too expensive

What does this MR do?

On the MR list page, we like to display how many MRs were found from the filtered search query in total. However, especially when the filter includes conditions on the MR title or description, this can be very expensive to calculate, and involve reading gigabytes of text data from the database.

As long as the data is already in the page cache, this usually finishes within the 15-second timeout on GitLab.com, but if the database cache is cold, a statement timeout is the usual occurrence.

More generally, it's not very clever to spend so much time calculating a piece of information with marginal value.

This MR applies a shorter limit to the counting statements and provides for graceful fallback to a '?' value, with a nice tooltip, if the query times out. This means we're able to view the results in a reasonable time, rather than the page taking a long time to load, or not loading at all.

Here's a patch that lets us easily test the impact in GDK:

diff --git a/app/finders/issuable_finder.rb b/app/finders/issuable_finder.rb
index 569a494e0a3..ae45dbec0ad 100644
--- a/app/finders/issuable_finder.rb
+++ b/app/finders/issuable_finder.rb
@@ -161,6 +161,7 @@ class IssuableFinder
   #
   # rubocop: disable CodeReuse/ActiveRecord
   def count_by_state
+    raise ActiveRecord::QueryCanceled, "TABLE FLIP"
     count_params = params.merge(state: nil, sort: nil, force_cte: true)
     finder = self.class.new(current_user, count_params)

With this applied, and the feature flag enabled, every list page should exhibit this behaviour.

Feature flag

Feature.enable(:soft_fail_count_by_state)

Rollout issue: #263222 (closed)

Screenshots

MR list, page 1, 1 page total

Issue list, page 1, 2 pages total

Issue list, page 2, 2 pages total

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team

Closes #249180 (closed)