Skip to content

fix: clean up data and manage runs during full data refresh in logical mode. New config option to skip full refresh during startup time: skipStartRefresh

Artyom Kartasov requested to merge logical-restore into master

Description

  • Remove all snapshots and PGDATA during a full refresh of data in a logical mode
  • Remove --clean and --if-exists flags for the pg_restore since PGDATA is going to be empty
  • Remove the forceInit options from logical configuration files, add a warning if PGDATA still contains any files or directories.
  • Add a new skipStartRefresh option for a logical mode to skip data refreshing

Will not be implemented:

  • db_name for pg_restore because we need to specify an existing database that we can connect to. Otherwise we will get the error:
[CONTAINER dblab_server]: 2023/03/06 17:01:01 restore.go:511: [INFO]   Running restore command for test 
[pg_restore --username postgres --dbname test --create --jobs 2 /var/lib/test/dblab_mount/test_dblab_pool/dump/test]

pg_restore: [archiver (db)] connection to database "test" failed: 
connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  database "test" does not exist

Related issue

Examples

forceInit deprecated

    logicalRestore:
      options:
        <<: *db_container
        dumpLocation: "/var/lib/dblab/dblab_pool/dump"
        parallelJobs: 4
        ignoreErrors: false
        <<: *db_configs

A new skipStartRefresh option:

retrieval:
  refresh:
    # Timetable is to be defined in crontab format: https://en.wikipedia.org/wiki/Cron#Overview
    timetable: "0 0 * * 1"

    # Skip data refresh while the retrieval starts.
    skipStartRefresh: false

Checklist

  • MR description has been reviewed
  • MR changes are functionally tested
  • MR does NOT have API/CLI changes OR there are API/CLI changes and they have been reviewed & DOCS ARE ADJUSTED (reference doc, etc)
  • MR does NOT have UI changes OR there are UI changes and they have been reviewed & UX IS REVIEWED
Edited by Nikolay Samokhvalov

Merge request reports