Move merge_request_diff_commits commit messages and trailers to object storage
In #331523 (comment 583654940) we found that a lot of storage in this table is used for storing commit messages and trailers. This data is never queried in a structured way, making it an excellent candidate to move to object storage. Combined with #331823 (closed) this would allow us to drastically reduce the size of this table.
The migration process would be more or less as follows:
- First, add a column to store whatever identifiers/paths/etc we need to determine where in object storage the data is stored. We'd do this for both commit messages and trailers
- We also need to add two columns to track the migrated state:
message_migratedandtrailers_migrated. Both are booleans that default tofalse - Add support to the code for reading from object storage if the data is present, falling back to the database data if nothing is found in object storage
- Add a background migration (scheduled using a post-deployment migration) that starts migrating these rows to object storage. When the data is migrated, the appropriate
X_migratedcolumn is set totrue - In a follow-up release we migrate any remaining data (= rows where
migratedis false) in-place. Depending on how fast the migration process takes, this may require more than just one release - In this same release we drop the old columns as well as the
migratedcolumns, and adjust the code to start reading from the new data only.
Edited by 🤖 GitLab Bot 🤖