"cryptsetup resize" hangs for several minutes

Issue description

Hello, I use the "LUKS on LVM" setup - a LUKS container is created for every individual LV. When I extend the volume, I encounter the situation when the extension of the LUKS device can take minutes. However, the extension of the underlying LV is always very quick. After digging into it, I observed that the "dm resume" stage hangs until all dirty pages are flushed. It can be a coincidence, but it's the only correlation I found. Also, the debug output says dm resume test-resize_crypt [ opencount flush ], which also mentions "flush".

Can you help me understand

Why does it take so long to resize?
Why is the "dm resume" required while the device is ACTIVE (not suspended)?
(assuming it's because of flush) Is this flush required? Can it be safely omitted?

Looking forward to your help. Thank you!

Steps for reproducing the issue

Create LV: lvcreate --size 400G -n test-resize vg-data
Create LUKS: echo -n 'test' | cryptsetup -v luksFormat --type luks2 /dev/vg-data/test-resize -
Open LUKS: echo -n 'test' | cryptsetup luksOpen /dev/vg-data/test-resize -
Create FS: mkfs.ext4 /dev/mapper/test-resize_crypt
Mount FS:

cd /tmp/
mkdir -p test-resize
mount /dev/mapper/test-resize_crypt test-resize

Then I run fio from inside a docker container with a mounted test-resize directory.

fio  --name=global -rw=randwrite --size=3g --name=job1 --numjobs 110  --time_based --runtime=10

After it finishes, there are a lot of dirty pages, and the kernel actively flushes them.

While the flush is happening, try to extend the volume:

Extend LV: lvextend --verbose --size +1G /dev/vg-data/test-resize
Extend LUKS: echo -n 'test' | cryptsetup -v --debug resize /dev/mapper/test-resize_crypt -

The last step always hangs after dm resume test-resize_crypt [ opencount flush ] [16384] (*1).

When I run the cryptsetup resize under strace --tT -ff, I can always see the same long-running syscall:

12:50:32 ioctl(3, DM_DEV_SUSPEND, {version=4.0.0, data_size=16384, name="test-resize_crypt", event_nr=6296210, flags=DM_EXISTS_FLAG} => {version=4.47.0, data_size=305, dev=makedev(0xfe, 0x1), name="test-resize_crypt", uuid="CRYPT-LUKS2-4018353329e44be09b21f54f74a2fd6f-test-resize_crypt", target_count=1, open_count=1, event_nr=0, flags=DM_EXISTS_FLAG|DM_ACTIVE_PRESENT_FLAG|DM_UEVENT_GENERATED_FLAG}) = 0 <303.679976>

The existing dirty pages I checked like this

grep Dirty /proc/meminfo

Additional info

# uname -a
Linux my-host 6.1.30-4 #1 SMP Wed Jul  5 22:36:53 UTC 2023 x86_64 GNU/Linux
#
# cryptsetup --version
cryptsetup 2.3.7
#
# dmsetup --version
Library version:   1.02.175 (2021-01-08)
Driver version:    4.47.0

I also tested this scenario on

Debian 5.10
crytpsetup 2.6.1
LUKS1 and LUKS2

The behavior is the same.

I also tested the volume extension without LUKS (LV + FS) - the extension is very quick.

Debug log

# echo -n 'test' | cryptsetup -v --debug resize  /dev/mapper/test-resize_crypt -
# cryptsetup 2.3.7 processing "cryptsetup -v --debug resize /dev/mapper/test-resize_crypt -"
# Running command resize.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating crypt device context by device /dev/mapper/test-resize_crypt.
# Initialising device-mapper backend library.
# dm version   [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-ioctl version 4.47.0.
# Detected dm-integrity version 1.10.0.
# Detected dm-crypt version 1.24.0.
# Device-mapper backend running with UDEV support enabled.
# dm status test-resize_crypt  [ opencount noflush ]   [16384] (*1)
# Releasing device-mapper backend.
# Trying to open and read device /dev/mapper/vg--data-test--resize with direct-io.
# Allocating context for crypt device /dev/mapper/vg--data-test--resize.
# Trying to open and read device /dev/mapper/vg--data-test--resize with direct-io.
# Initialising device-mapper backend library.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table test-resize_crypt  [ opencount flush securedata ]   [16384] (*1)
# dm status   (254:0) [ opencount noflush ]   [16384] (*1)
# Trying to open and read device /dev/mapper/vg--data-test--resize with direct-io.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm deps test-resize_crypt  [ opencount flush ]   [16384] (*1)
# dm table vg--data-test--resize  [ opencount flush securedata ]   [16384] (*1)
# Crypto backend (OpenSSL 1.1.1n  15 Mar 2022) initialized in cryptsetup library version 2.3.7.
# Detected kernel Linux 6.1.30-4 x86_64.
# Reloading LUKS2 header (repair disabled).
# Acquiring read lock for device /dev/mapper/vg--data-test--resize.
# Opening lock resource file /run/cryptsetup/L_254:0
# Verifying lock handle for /dev/mapper/vg--data-test--resize.
# Device /dev/mapper/vg--data-test--resize READ lock taken.
# Trying to read primary LUKS2 header at offset 0x0.
# Opening locked device /dev/mapper/vg--data-test--resize
# Veryfing locked device handle (bdev)
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:60af4ff804667ea245188ff933de6421e5e0f3c46ee83e14cb9fc27b73bcaf03 (on-disk)
# Checksum:60af4ff804667ea245188ff933de6421e5e0f3c46ee83e14cb9fc27b73bcaf03 (in-memory)
# Trying to read secondary LUKS2 header at offset 0x4000.
# Reusing open ro fd on device /dev/mapper/vg--data-test--resize
# LUKS2 header version 2 of size 16384 bytes, checksum sha256.
# Checksum:613a734c3d539f38c620935581981a08d88d9e2fd9dc5c78aac0bdf417a1d00c (on-disk)
# Checksum:613a734c3d539f38c620935581981a08d88d9e2fd9dc5c78aac0bdf417a1d00c (in-memory)
# Device size 432717955072, offset 16777216.
# Device /dev/mapper/vg--data-test--resize READ lock released.
# PBKDF argon2i, time_ms 2000 (iterations 0), max_memory_kb 1048576, parallel_threads 4.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table test-resize_crypt  [ opencount flush securedata ]   [16384] (*1)
# dm status   (254:0) [ opencount noflush ]   [16384] (*1)
# Trying to open and read device /dev/mapper/vg--data-test--resize with direct-io.
# Checking volume passphrase using token -1.
# STDIN descriptor passphrase entry requested.
# Checking volume passphrase [keyslot -1] using passphrase.
# Keyslot 0 priority 1 != 2 (required), skipped.
# Trying to open LUKS2 keyslot 0.
# Reading keyslot area [0x8000].
# Acquiring read lock for device /dev/mapper/vg--data-test--resize.
# Opening lock resource file /run/cryptsetup/L_254:0
# Verifying lock handle for /dev/mapper/vg--data-test--resize.
# Device /dev/mapper/vg--data-test--resize READ lock taken.
# Reusing open ro fd on device /dev/mapper/vg--data-test--resize
# Device /dev/mapper/vg--data-test--resize READ lock released.
# Verifying key from keyslot 0, digest 0.
# Loading key (64 bytes, type logon) in thread keyring.
Key slot 0 unlocked.
# Resizing device /dev/mapper/test-resize_crypt to 0 sectors.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table test-resize_crypt  [ opencount flush securedata ]   [16384] (*1)
# Calculated device size is 845119488 sectors (RW), offset 32768.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table test-resize_crypt  [ opencount flush securedata ]   [16384] (*1)
# dm status   (254:0) [ opencount noflush ]   [16384] (*1)
# Trying to open and read device /dev/mapper/vg--data-test--resize with direct-io.
# Calculated device size is 845119488 sectors (RW), offset 32768.
# dm reload test-resize_crypt  [ opencount flush securedata ]   [16384] (*1)
# Udev cookie 0xd4dfb68 (semid 32801) created
# Udev cookie 0xd4dfb68 (semid 32801) incremented to 1
# Udev cookie 0xd4dfb68 (semid 32801) incremented to 2
# Udev cookie 0xd4dfb68 (semid 32801) assigned to RESUME task(5) with flags DISABLE_LIBRARY_FALLBACK         (0x20)
# dm resume test-resize_crypt  [ opencount flush ]   [16384] (*1)


# test-resize_crypt: Stacking NODE_ADD (254,1) 0:6 0660 [trust_udev]
# Udev cookie 0xd4dfb68 (semid 32801) decremented to 1
# Udev cookie 0xd4dfb68 (semid 32801) waiting for zero
# Udev cookie 0xd4dfb68 (semid 32801) destroyed
# test-resize_crypt: Skipping NODE_ADD (254,1) 0:6 0660 [trust_udev]
# Releasing crypt device /dev/mapper/vg--data-test--resize context.
# Releasing device-mapper backend.
# Closing read only fd for /dev/mapper/vg--data-test--resize.
# Unlocking memory.
Command successful.

Edited Aug 27, 2023 by Ivan Shibitov