Add data source: pgBackRest
Goal
Add support pgBackRest as a data source for Database Lab Engine.
TODO / How to implement
Add to configuration the pgbackrest options
spec:
physicalRestore:
options:
tool: pgbackrest
envs:
PGBACKREST_REPO1_S3_KEY: "XXXXXXXXXXXXXXXXXX"
PGBACKREST_REPO1_S3_KEY_SECRET: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
pgbackrest:
stanza: my_stanza
repo1-type: s3
repo1-path: /pgbackrest
repo1-s3-endpoint: s3.amazonaws.com
repo1-s3-bucket: my_bucket
repo1-s3-region: us-east-1
process-max: 4
This is an example of configuration for s3 - AWS Simple Storage Service, parameters may differ, it can be a configuration for gcs - Google Cloud Storage or azure - Azure Blob Storage Service and etc. (https://pgbackrest.org/configuration.html#section-repository)
All specified parameters should be added as the command line parameters to execute pgbackrest restore command.
format:
pgbackrest { retrieval.spec.physicalRestore.options.pgbackrest } ... restore ... --recovery-option=restore_command='pgbackrest { retrieval.spec.physicalRestore.options.pgbackrest } ... archive-get %f "%p"'
example:
pgbackrest --stanza=my_stanza --repo1-type=s3 --repo1-path=/pgbackrest --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-bucket=my_bucket --repo1-s3-region=us-east-1 --process-max=4 --type=standby --pg1-path=${PGDATA} --delta restore --log-level-console=detail --recovery-option=restore_command='pgbackrest --stanza=my_stanza --repo1-type=s3 --repo1-path=/pgbackrest --repo1-s3-endpoint=s3.amazonaws.com --repo1-s3-bucket=my_bucket --repo1-s3-region=us-east-1 --process-max=4 --pg1-path=${PGDATA} archive-get %f "%p"'
Explanation of required parameters:
--pg1-path=${PGDATA}
and --recovery-option=restore_command
pgbackrest adds the restore_command parameter in the postgresql.auto.conf (or recovery.conf for PostgreSQL older than 12), and since the Postgres data directory path for Promote Instance is different, we override the restore_command specifying the variable ${PGDATA}
for the pg1-path
.
OR
DLE can control the PostgreSQL restore_command parameter, in which case we do not need to specify the --recovery-option parameter in the pgbackrest restore command.
--type=standby
pgbackrest will create a standby.signal
file. We can add this option as default, or if retrieval.spec.physicalRestore.options.sync.enabled: true
--delta restore
During a restore, by default the PostgreSQL data and tablespace directories are expected to be present but empty. This option performs a delta restore using checksums.
If the postgres data directory is not empty it will be overwritten. We can use the --delta parameter as default, or add a forceInit
parameter to the DLE configuration file.
--log-level-console=detail
Allows you to observe the restore process in the dblab_phr_xxxxxx container log: sudo docker logs --since 1m -f dblab_phr_xxxxxx
envs: PGBACKREST_REPO1_S3_KEY, PGBACKREST_REPO1_S3_KEY_SECRET
Option 'repo1-s3-key' is not allowed on the command-line. Specify the option in a environment variable instead.
Acceptance criteria
- pgBackRest package added to
postgresai/extended-postgres
docker image. - pgBackRest support is added in DLE.
- Added configuration example:
config.example.physical_pgbackrest.yml
- Added guide on the documentation page:
Data Source: pgBackRest