Store CI/CD job options/metadata in a separate table/database
Description
Our ci_builds
table is large. It not only has a lot of rows inside (20M on gitlab.com) but also needs better normalization.
We have a lot of duplication there, we also serialize some data inside options
column. Changing anything in ci_builds
table now on gitlab.com is really difficult because every dead tuple is large.
Proposal
Migrate build options / metadata to a separate table / database. Make it possible to define a retention policy that will remove old data that would result it not being able to retry a pipeline that is - for example - a year old. Consider moving this ephemeral data to a separate database, maybe some NoSQL database (we probably don't want to do that, but I'm mentioning this idea here, to not to forger about it).