Configuration

Config update required

• Changed idleTime option name to maxIdleMinutes (!87).
• Fixed bug when mountDir specified without trailing slash produced an error during thin-clone provision (!89, #108).

Client CLI

• Fixed --environment-id flag name (!88).

Documentation

• Added ZFS and LVM modes description to the README (!86).

CI

• Fixed CI rules for building images (!85).

Links

• Tutorial for using Database Lab with ZFS: https://postgres.ai/docs/database-lab/1_tutorial
• Tutorial for using Database Lab with LVM: https://postgres.ai/docs/database-lab/1_tutorial_lvm
• Database Lab open-source repository: https://gitlab.com/postgres-ai/database-lab
• Issue tracker (bugs reports, feature proposals are welcome): https://gitlab.com/postgres-ai/database-lab/issues
• SQL optimization usage example: https://gitlab.com/postgres-ai/joe
• Database changes verification usage example: https://gitlab.com/postgres-ai/ci-example
• Slack (English): https://database-lab-team-slack-invite.herokuapp.com/
• Telegram (Russian): https://t.me/databaselabru

Server

• Add support of the LVM2 volume manager in addition to the existing support of ZFS. Database Lab users can choose the means to enable thin cloning, between ZFS and LVM. Both thin-clone managers have pros and cons, the default is ZFS. The main difference is that ZFS supports thin-cloning based on various prepared snapshots of the file system (cloning from different database versions). The LVM module provides clones only based on the latest version of the database.
• Resetting a session is now much faster takes a few seconds: speed up clone destroying and resetting with --force flag which uses SIGKILL to terminate the main process inside of a container.
• Postgres configuration updates: load pg_stat_statement and auto_explain modules, avoid autovacuum running in the "transaction ID wraparound prevention" mode that may block user actions such as ALTER TABLE or CREATE INDEX.

Configuration

Config update required

• The mode value changed from zfs to local (currently, the only supported).
• A new mandatory option: thinCloneManager. Supported values: zfs, lvm.

API

• Added new struct CloneUpdateRequest for requests to update clones. When the PATCH method is used, the new struct is now used in the response.

Client CLI

• Allow choosing a snapshot when creating a clone.
• Increase the default polling timeout in the Database Lab client from 30s to 60s. This was done to mitigate slow reset (which is now significantly faster; it was decided to keep the timeout value increased).

CI

• Build containers for feature branches and push them to the GitLab container registry. Only the latest versions are to be stored.

Links

• 🆕 Tutorial for using Database Lab with LVM: https://postgres.ai/docs/database-lab/1_tutorial_lvm
• Tutorial for using Database Lab with ZFS: https://postgres.ai/docs/database-lab/1_tutorial
• Database Lab open-source repository: https://gitlab.com/postgres-ai/database-lab
• Issue tracker (bugs reports, feature proposals are welcome): https://gitlab.com/postgres-ai/database-lab/issues
• SQL optimization usage example: https://gitlab.com/postgres-ai/joe
• Database changes verification usage example: https://gitlab.com/postgres-ai/ci-example
• Slack (English): https://database-lab-team-slack-invite.herokuapp.com/
• Telegram (Russian): https://t.me/databaselabru

Bug fixes

• Always connect with dbname=postgres to detect idle clones.
• Do not ignore idle in transaction sessions when detecting idle clones.

Links

• Project repository: https://gitlab.com/postgres-ai/database-lab
• Issue tracker (for bugs reports, feature proposals): https://gitlab.com/postgres-ai/database-lab/issues
• SQL optimization usage example: https://gitlab.com/postgres-ai/joe
• Database changes verification usage example: https://gitlab.com/postgres-ai/ci-example
• Tutorial: https://postgres.ai/docs/database-lab/1_tutorial
• Slack (English): https://database-lab-team-slack-invite.herokuapp.com/
• Telegram (Russian): https://t.me/databaselabru

Server

• Fix: choose the latest snapshot by default on the clone creation. If multiple snapshots are available to be used in thin provisioning, and the snapshot parameter is not specified in the request for a new clone,
• Internal: Refactor some methods in the cloning service.
• Fix: Take into account active connections while determining idle clones. Previously, if query (such as CREATE INDEX ..) runs longer than "maximum allowed idle time", it got canceled and the thin clone was destroyed. Now both Postgres logs and pg_stat_activity are analyzed to determine if the clone is idle.
• Docs: add Swagger to describe API https://postgres.ai/swagger-ui/dblab/

Client CLI

• Fix: Add missing insecure parameter to init command.
• Internal: Add the debug flag.

CI

• Ignore Golang version tags (e.g. "v0.2.3") tags.
• Build containers both for server and client.
• Do not update the "latest" tag on release candidate versions (any tag with letters , e.g. "0.2rc").
• Moved cli-install script to /scripts directory.
• Build Swagger UI container on release tags.
• Parallel image building jobs.
• Removed useless cache which wasted time on downloading.

Links

• Project repository: https://gitlab.com/postgres-ai/database-lab
• Issue tracker (for bugs reports, feature proposals): https://gitlab.com/postgres-ai/database-lab/issues
• SQL optimization usage example: https://gitlab.com/postgres-ai/joe
• Database changes verification usage example: https://gitlab.com/postgres-ai/ci-example
• Tutorial: https://postgres.ai/docs/database-lab/1_tutorial
• Slack (English): https://database-lab-team-slack-invite.herokuapp.com/
• Telegram (Russian): https://t.me/databaselabru

Features

• CLI: Allow specifying "insecure" flag in the configuration for a particular Database Lab instance.

Bug fixes

• Skip session state error and use default clone size instead. In particular, this fixes the case when Joe bot gets errors when processing reset command.

Links

• Community Slack (English): https://database-lab-team-slack-invite.herokuapp.com/

Documentation

• Improved Swagger API description.

Bug fixes

• Fixed an empty struct for idle sessions.

Links

• Community Slack (English): https://database-lab-team-slack-invite.herokuapp.com/

In Database Lab 0.2.0, all the components now run in containers. To start using Database Lab is now only needed to install Docker and ZFS, everything else is inside containers. Customer PostgreSQL containers are supported.

Server

• All components of Database Lab server now run on Docker containers.
• Support custom Postgres containers.
• Ability to customize Postgres user used for management (postgres is default).
• Verification token flag changed from "-v" to "-t".
• Internal: use constants for response errors.
• Various improvements of the snapshotting script.

API

• Synchronous mode for clone methods (now default).
• Parameter name fully removed
• Parameter id can now be set by user (optionally).
• Internal: added Metadata to Clone model.

Client CLI

• Output improved.
• Show "unchanged" message in "config update" command if params match the current config.
• Added client installation script.

Documentation

• Added Swagger specification.
• Added Postman collection.
• Updated README.
• Tutorial fully reworked.
• New pages/sections in the docs: Overview, Workflow, Requirements, CLI Reference, Q&A.
• Added "ci-example" repository.

CI

• Build and push Docker images to Docker Hub and Gitlab Registry.
• Build and upload binaries for linux, alpine, darwin to GitLab.

Links

• Project repository: https://gitlab.com/postgres-ai/database-lab
• Issue tracker (for bugs reports, feature proposals): https://gitlab.com/postgres-ai/database-lab/issues
• SQL optimization usage example: https://gitlab.com/postgres-ai/joe
• Database changes verification usage example: https://gitlab.com/postgres-ai/ci-example
• Tutorial: https://postgres.ai/docs/database-lab/1_tutorial
• Slack (English): https://database-lab-team-slack-invite.herokuapp.com/
• Telegram (Russian): https://t.me/databaselabru

Features

• Superfast cloning of large databases.
• REST API.
• Client SDK.
• Client CLI tool.
• Auto-deletion of clones after some period of inactivity (by default, 2 hours).
• Protection from deletion: a clone can be marked as "protected", which means that it cannot be deleted, neither manually nor automatically.
• Postgres versions 9.6, 10, 11, 12 are supported.
• Postgres clones are running in Docker containers. Custom Docker images are supported, to allow additional extensions, libraries, custom Postgres builds.
• Custom postgresql.conf addition and pg_hba.conf replacement to control settings of Postgres clones.
• For security, all Postgres users are disabled, and a new one is created (configure at clone request time).

Known issues and future work

1. Currently, to install Database Lab, a lot of software needs to be installed: Docker, Go, ZFS, Postgres, more. See Tutorial. Tested on Ubuntu 18.04. In the upcoming Database Lab versions, we will move everything to containers, which means that only Docker will be a requirement, and any modern Linux will be supported.
2. If a Database Lab instance is restarted, all the clones get deleted. We plan to implement persistent clones in the future versions, to keep all existing clones during restarts.

Links

• Community Slack (English): https://database-lab-team-slack-invite.herokuapp.com/