Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
    • Switch to GitLab Next
  • Sign in / Register
Database Lab Engine – instant database clones to boost development
Database Lab Engine – instant database clones to boost development
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 63
    • Issues 63
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
    • Iterations
  • Merge Requests 3
    • Merge Requests 3
  • Requirements
    • Requirements
    • List
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Code Review
    • Insights
    • Issue
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Postgres.ai
  • Database Lab Engine – instant database clones to boost developmentDatabase Lab Engine – instant database clones to boost development
  • Issues
  • #231

Closed
Open
Opened Jan 27, 2021 by Artyom Kartasov@akartasovMaintainer

Failed clone cleanup

Goal

Fix the error in DLE logs:

2021/01/25 04:16:40 [ERROR] Failed to check the idleness of clone: failed to get the last session activity: failed to init file selector: failed to read a log directory: open /var/lib/dblab/clones/dblab_clone_6002/data/log: no such file or directory.

In dblab_server:


bash-5.0# ls -la /var/lib/dblab/clones/dblab_clone_6002
total 9
drwxr-xr-x    2 root     root             2 Dec 28 12:01 .
drwxr-xr-x  276 root     root           276 Jan 25 00:00 ..

In clone container:

ubuntu@db-clones:~/.dblab$ sudo docker exec -it dblab_clone_6002 bash
root@99c8113376eb:/# ls -la /var/lib/dblab/clones/dblab_clone_6002/data/log | head
total 21546
drwx------  2 postgres postgres   3092 Jan 25 04:00 .
drwx------ 21 postgres postgres     36 Jan 25 04:00 ..
-rwx------  1 postgres postgres   2256 Dec 24 11:17 postgresql-2020-12-24_111738.csv
-rwx------  1 postgres postgres    144 Dec 24 11:17 postgresql-2020-12-24_111738.log
-rwx------  1 postgres postgres   2226 Dec 24 11:17 postgresql-2020-12-24_111751.csv
-rwx------  1 postgres postgres    144 Dec 24 11:17 postgresql-2020-12-24_111751.log
-rwx------  1 postgres postgres   2218 Dec 24 11:18 postgresql-2020-12-24_111759.csv
-rwx------  1 postgres postgres    143 Dec 24 11:18 postgresql-2020-12-24_111759.log
-rwx------  1 postgres postgres   2226 Dec 24 11:18 postgresql-2020-12-24_111813.csv

It seems the reason is that the mount points have been removed while snapshots cleaned up

2021/01/03 18:00:00 [ERROR]  failed to clean up snapshots automatically: failed to clean up snapshots: failed to clean up snapshots: RunnerError(cmd="zfs list -t snapshot -H -o name -s dblab:datastateat 
-s creation -r dblab_pool | grep -v clone | head -n -4 | xargs -n1 --no-run-if-empty zfs destroy -R", inerr="exit status 123", stderr="umount: can't unmount /var/lib/dblab/clones/clone_pre_20201224121041
: Resource busy
cannot unmount '/var/lib/dblab/clones/clone_pre_20201224121041': umount failed
cannot destroy snapshot dblab_pool@snapshot_20201224121041_pre: snapshot is cloned
cannot destroy 'dblab_pool/dblab_clone_6000': dataset is busy
cannot destroy snapshot dblab_pool/clone_pre_20201226180000@snapshot_20201226175931: snapshot is cloned
cannot destroy snapshot dblab_pool@snapshot_20201226180000_pre: snapshot is cloned
cannot destroy 'dblab_pool/dblab_clone_6001': dataset is busy
cannot destroy snapshot dblab_pool/clone_pre_20201227120000@snapshot_20201227114009: snapshot is cloned
cannot destroy snapshot dblab_pool@snapshot_20201227120000_pre: snapshot is cloned
cannot destroy 'dblab_pool/dblab_clone_6002': dataset is busy
cannot destroy snapshot dblab_pool/clone_pre_20210102180000@snapshot_20210102175434: snapshot is cloned
cannot destroy snapshot dblab_pool@snapshot_20210102180000_pre: snapshot is cloned
" exit="123")

TODO / How to implement

  • avoid forced deletion
  • remove temporary created _pre snapshots and clones

Acceptance criteria

Assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
None
Due date
None
Reference: postgres-ai/database-lab#231