allowlist and disallowlist for schemas: add a configuration option to dump schemas and exclude schemas for data retrieval
Goal
Add a configuration option to dump schemas and exclude schemas for logical dump.
TODO / How to implement
Add a new configuration options (schemas
) and (excludeSchemas
) to the section retrieval.spec.logicalDump.options.databases
:
retrieval:
spec:
logicalDump:
options:
# Option for specifying the database list that must be copied.
# By default, DLE dumps and restores all available databases.
# Do not specify the databases section to take all databases.
databases:
database1:
# Options for a partial dump.
# Do not specify the tables section to dump all available tables.
# tables:
# - table1
# Dump only schemas matching pattern; this selects both the schema itself, and all its contained objects.
# Do not specify the schemas section to dump all available schemas.
schemas:
- schema1
# Do not dump any schemas matching pattern.
excludeSchemas:
- schema2
-
Add the option to the pg_dump command (the
logicalDump
stage) with the flag--schema=pattern
for each schema/pattern in the "schemas
" list. -
Add the option to the pg_dump command (the
logicalDump
stage) with the flag--exclude-schema=pattern
for each schema/pattern in the "excludeSchemas
" list.- Applies globally for all databases if "schemas" and "excludeSchemas" is specified in the "databases" section or for a specific database if is specified in the "databases.databasename" section.
Example (for 'schemas'):
pg_dump --create --host <HOST> --port <PORT> --username <USERNAME> --dbname database1 --schema='schema1' --jobs 2 --format directory --file /var/lib/dblab/dblab_pool/dump/database1
Example (for 'excludeSchemas'):
pg_dump --create --host <HOST> --port <PORT> --username <USERNAME> --dbname database1 --exclude-schema='schema2' --jobs 2 --format directory --file /var/lib/dblab/dblab_pool/dump/database1
- See more details about the flag in the official docs: https://www.postgresql.org/docs/current/app-pgdump.html
-
Add tests (unit and integration) to verify the correctness of custom data retrieving
-
Describe new options in examples of configuration files (https://gitlab.com/postgres-ai/database-lab/-/tree/master/engine/configs) and in the official docs (https://gitlab.com/postgres-ai/docs)
By default, the new options should be opt-out (commented out).
Acceptance criteria
- User can dump specified schemas.
- User can exclude specified schemas. The rest of the schemas are dumped as usual.
- Consider the case when both options(
schemas
andexcludeSchemas
) are specified at once - New tests covers all possible cases of retrieving data
- Options are described in the docs