Table disallowlist: add a configuration option to exclude tables for data retrieval
Goal
At the moment there are no options to exclude a single table during logical dump, user can only specify a list of tables to copy.
TODO / How to implement
Add a new configuration option (excludeTables
) to the section retrieval.spec.logicalDump.options.databases
:
retrieval:
spec:
logicalDump:
options:
# Option for specifying the database list that must be copied.
# By default, DLE dumps and restores all available databases.
# Do not specify the databases section to take all databases.
databases:
database1:
# Options for a partial dump.
# Do not specify the tables section to dump all available tables.
# tables:
# - table1
# Do not dump any tables matching pattern.
# Accept specific schemas and tables or will allow for wildcards (*) for more flexibility.
excludeTables:
- table2
- tableN
-
Add the table specified option to the pg_dump command (the
logicalDump
stage) with the flag--exclude-table=pattern
for each table/pattern in the "excludeTables
" list.- Applies globally for all databases if "excludeTables" is specified in the "databases" section or for a specific database if "excludeTables" is specified for a specific database in the "databases.databasename" section.
Example:
pg_dump --create --host <HOST> --port <PORT> --username <USERNAME> --dbname database1 --jobs 2 --format directory --exclude-table='table2' --exclude-table='tableN' --file /var/lib/dblab/dblab_pool/dump/database1
- See more details about the flag in the official docs: https://www.postgresql.org/docs/current/app-pgdump.html
-
Add tests (unit and integration) to verify the correctness of custom data retrieving
-
Describe new options in examples of configuration files (https://gitlab.com/postgres-ai/database-lab/-/tree/master/engine/configs) and in the official docs (https://gitlab.com/postgres-ai/docs)
By default, the new options should be opt-out (commented out).
Acceptance criteria
- User can exclude specified tables.
- The rest of the tables are dumped as usual.
- Consider the case when both options(
tables
andexcludeTables
) are specified at once - New tests covers all possible cases of retrieving data
- Options are described in the docs
Edited by Vitaliy Kukharik