Skip to content

Detect and log cross-database modifications in production (no `QueryAnalyzer`)

What does this MR do and why?

This was extracted from !73316 (closed) since that went in a lot of different directions and I wanted to keep a working MR as a backup in case we get stuck with some of the various refactorings going on with that. This MR is not perfect and a little clumsy but it is tested and works and doesn't introduce serious regressions (to my knowledge). It also has some easy ways to roll back if there are issues.

This adds a Rack middleware and Sidekiq middleware for detecting "Cross-database modification" violations. These occur when you query 2 different databases in the context of a single transaction. Previously we only wrapped our RSpec code with this detection logic but we found that it was often causing lots of false positives in RSpec (in test-only code) which made it hard to sort through the real issues. In addition we want extra validation by running this in production and therefore detecting code that our specs may be missing.

The middlewares use a feature flag detect_cross_database_modification which we intend to enable for a small percentage of time.

The middlewares can also be disabled by setting the env var DISABLE_CROSS_DATABASE_MODIFICATION_DETECTION=true and restarting the application. This environment variable is an extra safety net if the middlewares are causing problems.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #341783 (closed)

Edited by Dylan Griffith

Merge request reports