Prevent markdown version changes from impacting GitLab.com DB
During the incident gitlab-com/gl-infra/production#4481 (comment 568197343), we realized that changing markdown version numbers puts a lot of additional strain on the GitLab.com database.
This is because every row with cached markdown on it needs to be updated.
Unfortunately some of these tables, such as namespace
are critical to the performance of GitLab.com. The updates lead to additional dead tuples, which slows things down.
We need to split the "business-end" of these performance critical tables from the UI. We have two options:
- Wait until the stability of the database has improved - from the traversal_ids effort and the effort to break up ci_builds and then remove the lock
- Remove cached_markdown fields from certain key tables, and move them into 1:1 mapped tables that don't get used in critical queries (specifically TheBigQuery)
Note that we're still seeing elevated dead tuples, several days after this change was made.
DONE In the mean time, @engwan suggested we lock the Ruby file containing the current markdown version number, to avoid unnecessary changes here.
Links
Some of the signals that could indicate that things are "good enough" to remove the lock might include:
- Fewer alerts for cascading tuple events on the
namespaces
table: https://gitlab.slack.com/archives/C01RUV22QRJ/p1621342016000900 - Patroni SLI improvements: https://dashboards.gitlab.net/d/patroni-main/patroni-overview?viewPanel=3543037459&orgId=1&from=now-7d&to=now (this is already looking a lot better than when this issue was raised)