luksHeaderBackup normally fails if mlockall succeeds

Issue description

The Linux kernel's default RLIMIT_MEMLOCK (ulimit -l) used to be 64 KiB, but it was raised to 8 MiB in 5.16. With the 64 KiB limit, I believe mlockall would always fail. Therefore, later allocations, such as the fairly large buffer allocated by luksHeaderBackup, would not be subject to the locked memory limit.

With the new 8 MiB limit, it appears that mlockall can succeed in certain cases. If so, when luksHeaderBackup tries to allocate, it hits the locked memory limit and the command fails. Whether or not mlockall succeeds seems to be highly dependent on the test setup. I can reliably trigger the issue while running cryptsetup test suite on an armv7l machine. Strangely, I can't trigger it I run the same command from the test suite outside of the test script.

It seems like cryptsetup is trying to lock an excessive amount of memory, and some commands can only work if the locking fails (or presumably the limit is set very high).

Additional info

All of my testing occurred on NixOS

Debug log

armv7l inside test script; 8 MiB limit; mlockall succeeds, therefore command fails:

# cryptsetup 2.4.3 processing "/build/cryptsetup-2.4.3/.libs/lt-cryptsetup --debug luksHeaderBackup luks-test --header-backup-file luks-header"
# Running command luksHeaderBackup.
# Locking memory.
# setpriority -18 failed: Permission denied
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device luks-test.
# Trying to open and read device luks-test with direct-io.
# Initialising device-mapper backend library.
# Crypto backend (OpenSSL 1.1.1o  3 May 2022) initialized in cryptsetup library version 2.4.3.
# Detected kernel Linux 5.16.15 armv7l.
# PBKDF pbkdf2-sha256, time_ms 2000 (iterations 0).
# Reading LUKS header of size 1024 from device luks-test
# Key length 64, device size 20000 sectors, header size 4036 sectors.
# Requested header backup of device luks-test (any type) to file luks-header.
# Reading LUKS header of size 1024 from device luks-test
# Reusing open ro fd on device luks-test
# Key length 64, device size 20000 sectors, header size 4036 sectors.
# Releasing crypt device luks-test context.
# Releasing device-mapper backend.
# Closing read only fd for luks-test.
# Unlocking memory.
Command failed with code -3 (out of memory).

armv7l outside test script; 8 MiB limit; mlockall fails, therefore command succeeds:

# cryptsetup 2.4.3 processing "/build/cryptsetup-2.4.3/.libs/lt-cryptsetup --debug luksHeaderBackup luks-test --header-backup-file luks-header"
# Running command luksHeaderBackup.
# Locking memory.
# Cannot lock memory with mlockall.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device luks-test.
# Trying to open and read device luks-test with direct-io.
# Initialising device-mapper backend library.
# Crypto backend (OpenSSL 1.1.1o  3 May 2022) initialized in cryptsetup library version 2.4.3.
# Detected kernel Linux 5.16.15 armv7l.
# PBKDF pbkdf2-sha256, time_ms 2000 (iterations 0).
# Reading LUKS header of size 1024 from device luks-test
# Key length 64, device size 20000 sectors, header size 4036 sectors.
# Requested header backup of device luks-test (any type) to file luks-header.
# Reading LUKS header of size 1024 from device luks-test
# Reusing open ro fd on device luks-test
# Key length 64, device size 20000 sectors, header size 4036 sectors.
# Storing backup of header (1024 bytes) and keyslot area (2062336 bytes).
# Output backup file size: 2068480 bytes.
# Reusing open ro fd on device luks-test
# Releasing crypt device luks-test context.
# Releasing device-mapper backend.
# Closing read only fd for luks-test.
Command successful.

x86_64; 64 KiB limit; mlockall fails, therefore command succeeds:

# cryptsetup 2.4.3 processing "/build/cryptsetup-2.4.3/.libs/lt-cryptsetup --debug luksHeaderBackup luks-test --header-backup-file luks-header"
# Running command luksHeaderBackup.
# Locking memory.
# Cannot lock memory with mlockall.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device luks-test.
# Trying to open and read device luks-test with direct-io.
# Initialising device-mapper backend library.
# Crypto backend (OpenSSL 1.1.1o  3 May 2022) initialized in cryptsetup library version 2.4.3.
# Detected kernel Linux 5.15.28 x86_64.
# PBKDF pbkdf2-sha256, time_ms 2000 (iterations 0).
# Reading LUKS header of size 1024 from device luks-test
# Key length 64, device size 20000 sectors, header size 4036 sectors.
# Requested header backup of device luks-test (any type) to file luks-header.
# Reading LUKS header of size 1024 from device luks-test
# Reusing open ro fd on device luks-test
# Key length 64, device size 20000 sectors, header size 4036 sectors.
# Storing backup of header (1024 bytes) and keyslot area (2062336 bytes).
# Output backup file size: 2068480 bytes.
# Reusing open ro fd on device luks-test
# Releasing crypt device luks-test context.
# Releasing device-mapper backend.
# Closing read only fd for luks-test.
Command successful.