cryptsetup luksFormat slowdown of factor ~ 6 when using hardened memory allocator Hardened Malloc (#617) · Issues · cryptsetup / cryptsetup

cryptsetup luksFormat slowdown of factor ~ 6 when using hardened memory allocator Hardened Malloc

Issue description

cryptsetup luksFormat gets slower by approximately factor ~ 6 when using hardened memory allocator Hardened Malloc.

Without Hardened Malloc:

time sudo ./test 
+ set -e
+ rm -f /tmp/test
+ dd if=/dev/zero bs=1M count=50 of=/tmp/test
50+0 records in
50+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 0.0258572 s, 2.0 GB/s
+ echo test
+ cryptsetup --batch-mode luksFormat /tmp/test

real	0m7.925s
user	0m10.994s
sys	0m0.867s

With Hardened Malloc:

time sudo ./test
+ set -e
+ rm -f /tmp/test
+ dd if=/dev/zero bs=1M count=50 of=/tmp/test
50+0 records in
50+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 0.0261877 s, 2.0 GB/s
+ cryptsetup --batch-mode luksFormat /tmp/test
+ echo test

real	0m40.255s
user	0m9.943s
sys	0m33.331s

Steps for reproducing the issue

run some benchmark using cryptsetup luksFormat without hardened malloc installed
install hardened malloc
re-run benchmark
compare benchmarks

Additional info

Debian buster

My test script.

#!/bin/bash
set -x
set -e

rm -f /tmp/test

dd if=/dev/zero bs=1M count=50 of=/tmp/test

echo test | cryptsetup --debug --batch-mode luksFormat /tmp/test

//cc @thestinger, developer of Hardened Malloc

Debug log

Without hardened malloc.

+ set -e
+ rm -f /tmp/test
+ dd if=/dev/zero bs=1M count=50 of=/tmp/test
50+0 records in
50+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 0.0256035 s, 2.0 GB/s
+ echo test
+ cryptsetup --debug --batch-mode luksFormat /tmp/test
# cryptsetup 2.1.0 processing "cryptsetup --debug --batch-mode luksFormat /tmp/test"
# Running command luksFormat.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /tmp/test.
# Trying to open and read device /tmp/test with direct-io.
# Initialising device-mapper backend library.
# STDIN descriptor passphrase entry requested.
# Crypto backend (OpenSSL 1.1.1d  10 Sep 2019) initialized in cryptsetup library version 2.1.0.
# Detected kernel Linux 4.19.125-1.pvops.qubes.x86_64 x86_64.
# Only 2 active CPUs detected, PBKDF threads decreased from 4 to 2.
# Not enough physical memory detected, PBKDF max memory decreased from 1048576kB to 339862kB.
# PBKDF argon2i, hash sha256, time_ms 2000 (iterations 0), max_memory_kb 339862, parallel_threads 2.
# Formatting device /tmp/test as type LUKS2.
# Topology info for /tmp/test not supported, using default offset 1048576 bytes.
# Checking if cipher aes-xts-plain64 is usable.
# Using userspace crypto wrapper to access keyslot area.
# Formatting LUKS2 with JSON metadata area 12288 bytes and keyslots area 16744448 bytes.
# Creating new digest 0 (pbkdf2).
# Setting PBKDF2 type key digest 0.
# Running pbkdf2(sha256) benchmark.
# PBKDF benchmark: memory cost = 0, iterations = 840205, threads = 0 (took 39 ms)
# PBKDF benchmark: memory cost = 0, iterations = 846991, threads = 0 (took 619 ms)
# Benchmark returns pbkdf2(sha256) 846991 iterations, 0 memory, 0 threads (for 512-bits key).
# Segment 0 assigned to digest 0.
# Wiping LUKS areas (0x000000 - 0x1000000) with zeroes.
# Wiping keyslots area (0x008000 - 0x1000000) with random data.
# Device size 52428800, offset 16777216.
# Acquiring write lock for device /tmp/test.
# Verifying write lock handle for device /tmp/test.
# Device /tmp/test WRITE lock taken.
# Trying to write LUKS2 header (16384 bytes) at offset 0.
# Opening locked device /tmp/test
# Veryfing locked device handle (regular file)
# Checksum:5929c7ee2ee79c2a4e204bab27891c5e1e6104e84c12d12506871c2ac16d2306 (in-memory)
# Trying to write LUKS2 header (16384 bytes) at offset 16384.
# Opening locked device /tmp/test
# Veryfing locked device handle (regular file)
# Checksum:8c04b8fd64c488e4f1725cedca059f8b10e4ac434204e233a73fa691e25b79a4 (in-memory)
# Device /tmp/test WRITE lock released.
# Adding new keyslot -1 using volume key.
# Adding new keyslot -1 with volume key assigned to a crypt segment.
# Selected keyslot 0.
# Verifying key digest 0.
# Keyslot 0 assigned to digest 0.
# Trying to allocate LUKS2 keyslot 0.
# Found area 32768 -> 290816
# Running argon2i() benchmark.
# PBKDF benchmark: memory cost = 32, iterations = 4, threads = 2 (took 5 ms)
# PBKDF benchmark: memory cost = 512, iterations = 4, threads = 2 (took 1 ms)
# PBKDF benchmark: memory cost = 8192, iterations = 4, threads = 2 (took 29 ms)
# PBKDF benchmark: memory cost = 70620, iterations = 4, threads = 2 (took 201 ms)
# PBKDF benchmark: memory cost = 87835, iterations = 4, threads = 2 (took 194 ms)
# PBKDF benchmark: memory cost = 113189, iterations = 4, threads = 2 (took 259 ms)
# PBKDF benchmark: memory cost = 339862, iterations = 10, threads = 2 (took 1976 ms)
# Benchmark returns argon2i() 10 iterations, 339862 memory, 2 threads (for 512-bits key).
# Calculating attributes for LUKS2 keyslot 0.
# Updating keyslot area [0x8000].
# Acquiring write lock for device /tmp/test.
# Verifying write lock handle for device /tmp/test.
# Device /tmp/test WRITE lock taken.
# Opening locked device /tmp/test
# Veryfing locked device handle (regular file)
# Device /tmp/test WRITE lock released.
# Device size 52428800, offset 16777216.
# Acquiring write lock for device /tmp/test.
# Verifying write lock handle for device /tmp/test.
# Device /tmp/test WRITE lock taken.
# Trying to write LUKS2 header (16384 bytes) at offset 0.
# Opening locked device /tmp/test
# Veryfing locked device handle (regular file)
# Checksum:e45303cf0534fbf492aaa8ebf0d92c106b21b2e64024f36ddb6163cde5e75ad4 (in-memory)
# Trying to write LUKS2 header (16384 bytes) at offset 16384.
# Opening locked device /tmp/test
# Veryfing locked device handle (regular file)
# Checksum:f6353f870a36ecb747a37e65181d861bd565b8b6aba036f867e7786e793f003e (in-memory)
# Device /tmp/test WRITE lock released.
Key slot 0 created.
# Releasing crypt device /tmp/test context.
# Releasing device-mapper backend.
# Unlocking memory.
Command successful.

real	0m7.581s
user	0m9.869s
sys	0m0.930s

With hardened malloc.

time sudo ./test
+ set -e
+ rm -f /tmp/test
+ dd if=/dev/zero bs=1M count=50 of=/tmp/test
50+0 records in
50+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 0.0251046 s, 2.1 GB/s
+ echo test
+ cryptsetup --debug --batch-mode luksFormat /tmp/test
# cryptsetup 2.1.0 processing "cryptsetup --debug --batch-mode luksFormat /tmp/test"
# Running command luksFormat.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating context for crypt device /tmp/test.
# Trying to open and read device /tmp/test with direct-io.
# Initialising device-mapper backend library.
# STDIN descriptor passphrase entry requested.
# Crypto backend (OpenSSL 1.1.1d  10 Sep 2019) initialized in cryptsetup library version 2.1.0.
# Detected kernel Linux 4.19.125-1.pvops.qubes.x86_64 x86_64.
# Only 2 active CPUs detected, PBKDF threads decreased from 4 to 2.
# Not enough physical memory detected, PBKDF max memory decreased from 1048576kB to 341210kB.
# PBKDF argon2i, hash sha256, time_ms 2000 (iterations 0), max_memory_kb 341210, parallel_threads 2.
# Formatting device /tmp/test as type LUKS2.
# Topology info for /tmp/test not supported, using default offset 1048576 bytes.
# Checking if cipher aes-xts-plain64 is usable.
# Using userspace crypto wrapper to access keyslot area.
# Formatting LUKS2 with JSON metadata area 12288 bytes and keyslots area 16744448 bytes.
# Creating new digest 0 (pbkdf2).
# Setting PBKDF2 type key digest 0.
# Running pbkdf2(sha256) benchmark.
# PBKDF benchmark: memory cost = 0, iterations = 840205, threads = 0 (took 39 ms)
# PBKDF benchmark: memory cost = 0, iterations = 852500, threads = 0 (took 615 ms)
# Benchmark returns pbkdf2(sha256) 852500 iterations, 0 memory, 0 threads (for 512-bits key).
# Segment 0 assigned to digest 0.
# Wiping LUKS areas (0x000000 - 0x1000000) with zeroes.
# Wiping keyslots area (0x008000 - 0x1000000) with random data.
# Device size 52428800, offset 16777216.
# Acquiring write lock for device /tmp/test.
# Verifying write lock handle for device /tmp/test.
# Device /tmp/test WRITE lock taken.
# Trying to write LUKS2 header (16384 bytes) at offset 0.
# Opening locked device /tmp/test
# Veryfing locked device handle (regular file)
# Checksum:d0ecc71988d93df516431a9babe6ba2e1d98bcb8163082f83f0eab25b97d9206 (in-memory)
# Trying to write LUKS2 header (16384 bytes) at offset 16384.
# Opening locked device /tmp/test
# Veryfing locked device handle (regular file)
# Checksum:3874184ad27bd0cdc48e1688c973a68d68fc6b3b380c03ee1b9c95ab10173329 (in-memory)
# Device /tmp/test WRITE lock released.
# Adding new keyslot -1 using volume key.
# Adding new keyslot -1 with volume key assigned to a crypt segment.
# Selected keyslot 0.
# Verifying key digest 0.
# Keyslot 0 assigned to digest 0.
# Trying to allocate LUKS2 keyslot 0.
# Found area 32768 -> 290816
# Running argon2i() benchmark.
# PBKDF benchmark: memory cost = 32, iterations = 4, threads = 2 (took 4 ms)
# PBKDF benchmark: memory cost = 512, iterations = 4, threads = 2 (took 1 ms)
# PBKDF benchmark: memory cost = 8192, iterations = 4, threads = 2 (took 21 ms)
# PBKDF benchmark: memory cost = 97523, iterations = 4, threads = 2 (took 204 ms)
# PBKDF benchmark: memory cost = 119513, iterations = 4, threads = 2 (took 269 ms)
# PBKDF benchmark: memory cost = 341210, iterations = 10, threads = 2 (took 2092 ms)
# Benchmark returns argon2i() 10 iterations, 341210 memory, 2 threads (for 512-bits key).
# Calculating attributes for LUKS2 keyslot 0.
# Updating keyslot area [0x8000].
# Acquiring write lock for device /tmp/test.
# Verifying write lock handle for device /tmp/test.
# Device /tmp/test WRITE lock taken.
# Opening locked device /tmp/test
# Veryfing locked device handle (regular file)
# Device /tmp/test WRITE lock released.
# Device size 52428800, offset 16777216.
# Acquiring write lock for device /tmp/test.
# Verifying write lock handle for device /tmp/test.
# Device /tmp/test WRITE lock taken.
# Trying to write LUKS2 header (16384 bytes) at offset 0.
# Opening locked device /tmp/test
# Veryfing locked device handle (regular file)
# Checksum:a280edc63729a53b1fffa92d6a8e3e73ad957ec6a3b3931206bab0f9435b3e6e (in-memory)
# Trying to write LUKS2 header (16384 bytes) at offset 16384.
# Opening locked device /tmp/test
# Veryfing locked device handle (regular file)
# Checksum:501335fa4afcef24700fbeabb4089ad9fc0fd53ae5fe6273a69210e6c037c148 (in-memory)
# Device /tmp/test WRITE lock released.
Key slot 0 created.
# Releasing crypt device /tmp/test context.
# Releasing device-mapper backend.
# Unlocking memory.
Command successful.

real	0m39.775s
user	0m9.513s
sys	0m32.816s