Make it easy to mutate data on-demand
Problem to solve
We recently faced a problem of filling
This provides a major performance improvement, and simplifies pipeline processing when used.
However, we faced a challenge that for all existing data this column is
and we don't really want to retroactively migrate all data.
I proposed that we start versioning pipeline object versions, allowing us to easily mutate the data to latest expected version in critical paths.
module Migratable column_name = :schema_version end class Pipeline include Migratable schema_migration 1, :update_scheduling_type! def update_scheduling_type! ... end end class ProcessPipelineService def execute pipeline.migrate! end end
- define a schema migration of version
1(the lowest number defined),
- we would run the migration on demand by calling a method
- if data is up-to date, we don't need to do anything,
- each table with
This allows us to:
- To see exactly the data migrations,
- Allow to easily migrate data between versions (only forward migration),
- Support well the migration of data for import/export, as data can be migrated on a go if needed,
- Overall reduce a complexity, as we do not need to do additional conditionals to support more ifs, but rather we can depend on a fact that data are migrated, and have an efficient way to not do it at all.