Creating and restoring database snapshots
It must be possible for a user to create a snapshot of a database. After creating a snapshot, a complete copy of the database must be stored somewhere. The user must be able to restore snapshot, when doing so, a copy of the snapshot must be created. The original database must continue to exist after the restoring.
Requirements
- There must be a configurable limit of how many snapshots can be created.
- There must be a configurable limit when the snapshots should be deleted automatically.
- The user files do not have to be stored in the snapshot because they already exist in the snapshot.
- It must be possible to restore the database into another group.
Questions
- What should happen when creating a snapshot while other users are editing the data?
- Should the snapshot be stored in the exportable JSON format?
- Where are we going to store the snapshots (database, file system)?
- Currently we already have to possibility to create a copy of groups, databases and tables by exporting and importing them. It will result in a JSON serializable dict that can also be imported again. It will result in a copy when doing so. This is the technology that's also used to create and install templates. At the moment it can only be used via the
export_group_applications
andimport_group_applications
managements commands. We might want to use this technology for the snapshots as well.
- Currently we already have to possibility to create a copy of groups, databases and tables by exporting and importing them. It will result in a JSON serializable dict that can also be imported again. It will result in a copy when doing so. This is the technology that's also used to create and install templates. At the moment it can only be used via the
Plan
- The snapshot implementation will be a
core
feature that will be available for any type of Baserow application - The create/restore snapshot will be based on the current import/export functionality that retrieves and serializes data to JSON and back, with the exception of skipping the retrieving and saving files as current files are already stored and not deleted
- Necessary changes for skipping file processing will become available through https://gitlab.com/bramw/baserow/-/merge_requests/848 MR
- Each application snapshot will be stored as a normal application in the database, but won't be connected to any
Group
so that no API user will be able to manipulate with the snapshot
- Create and restore will be implemented as async Celery tasks via the existing async job system
- This MR will provide implementation for
database
application
- This MR will provide implementation for
- Each
ApplicationType
can provide one specificJob
for creating a snapshot and oneJob
for restoring it - All snapshots will be stored in a separate
snapshots
table, providing snapshotname
,created by
,created at
, and alink to the application
the snapshot is for, andlink to the application
that was created as a snapshot- Snapshot
name
must be unique per application
- Snapshot
- Additionally, each snapshot will have a link to the async job doing the snapshot creation. This will help to only show snapshots in the UI that are already created
- Additionally, each snapshot will have many to many relationship with "restore" jobs, necessary to keep track the restoring progress.
- Snapshots that are being restored will not be able to be deleted in the meantime.
- Old snapshots will be deleted through a separate periodic async job .
- There will be an env var for configuring the maximum number of snapshots per group.
- There will be an env var for configuring the period of automatic snapshot deletion.
Edited by Petr Stribny