Introduce functionality for keeping track of datamodel versions, changes, and migrations
Summary
In case of changes to datamodels, it can be desirable to keep track of which state your database is in, which state is required for certain scripts, crawlers, ... to function, and what migrations might be necessary to come from one state to another. This can be obtained in various ways, one standard way for which we will implement helper functions, can be to keep track of (parts of) a datamodel in special Datamodel Records:
Datamodel:
obligatory_properties:
datamodel_version:
datatype: TEXT
recommended_properties:
datamodel_entities:
datatype: REFERENCE
where a datamodel record has a name and a version string, and optionally references to all entities defining this (part of a) datamodel. These references can even be to specific versions so that a sanity check can be performed whether the current HEAD of the referenced entities is still compatible.
Scripts can then check for the presence of a datamodel in a specific version, migrations can define transitions between one version and another, and it can be assured that they are not run twice or on the wrong version. The version can be oriented on semantic versioning in that additive changes to a datamodel (+ property/RT, ehat eas valid before remains valid afterwards) increase the minor version, and breaking changes (e.g., author property was string before, is now reference) increase the major version.
For this, we need to
-
Define the names for the record type and the properties in the above example -
add helper functions to this library for -
checking the presence and the version of a datamodel -
checking whether a script is applicable (only if datamodel A is present in version >1.1) -
book keeping for migrations: has been run, run all, ...
-
-
extend the model parser s.th. it creates a datamodel record with version and references automatically
There is a lot of room for future enhancements here, but we believe that this will define a good start which is reachable in relatively short time.