Skip to content

WIP: Introduce a CQRS mechanism for counter model attributes

Fabio Pitino requested to merge efficient-counters into master

What does this MR do?

This is currently a PoC that displays how we could solve the contention of updates on columns that are update-heavy. E.g. project_statistics table.

The problem is largely defined and discussed in #24469 (closed)

In this MR we introduce

  1. a new module that brings counter attributes functionality and makes a model attribute ready-only while scheduling asynchronous updates
  2. a background worker responsible for consolidating the pending updates
  3. an arbitrary attribute from ProjectStatistics to be converted to an "efficient counter"

Related issue: #24469 (closed)

What is the goal of this MR?

  • Ensure the direction is correct
  • Check what is missing from database efficiency point of view
  • Validate that the design can actually solve the problem

Design

In order to split writes from reads we can introduce an events/logs table for the model that contains efficient counters. Any time we want to update the specific "counter attribute" we write these events into a <model>_events table. Then the update will schedule a background job that is responsible for taking all the pending events and update the attribute in the main table. The read strategy should be completely different than the write strategy.

Does this MR meet the acceptance criteria?

Conformity

Edited by Fabio Pitino

Merge request reports