Python ctypes.CDLL() and ctypes.cdll.LoadLibrary() fail as the runtime linker relies on LD_LIBRARY_PATH

Issue

When trying to install PyTables, I ran into the following issue:

  * Found HDF5 headers at ``/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/HDF5/1.14.3-gompi-2023b/include``, library at ``/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/HDF5/1.14.3-gompi-2023b/lib``.
  .. WARNING:: Could not find the HDF5 runtime.
     The HDF5 shared library was *not* found in the default library
     paths. In case of runtime problems, please remember to install it.
  * Found LZO 2 headers at ``/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/LZO/2.10-GCCcore-13.2.0/include``, the library is located in the standard system search dirs.
  .. WARNING:: Could not find the LZO 2 runtime.
     The LZO 2 shared library was *not* found in the default library
     paths. In case of runtime problems, please remember to install it.
  * Skipping detection of LZO 1 since LZO 2 has already been found.
  * Found bzip2 headers at ``/usr/include``, library at ``/usr/lib64``.
  * Found blosc headers at ``/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Blosc/1.21.5-GCCcore-13.2.0/include``, the library is located in the standard system search dirs.
  .. WARNING:: Could not find the blosc runtime.
     The blosc shared library was *not* found in the default library
     paths. In case of runtime problems, please remember to install it.
  * Run 'blosc2_find_directories_hook'
  * Found blosc2 headers at ``/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Blosc2/2.13.2-GCCcore-13.2.0/include``, library at ``/home/casparl/eessi/versions/2023.06/software/linux/x86_64/amd/zen2/software/Blosc2/2.13.2-GCCcore-13.2.0/lib``.
    * Copying blosc2 runtime library to 'tables' dir because it was not found in standard locations
  Traceback (most recent call last):
    File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/site-packages/setuptools/build_meta.py", line 434, in build_wheel
      return self._build_with_temp_dir(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/site-packages/setuptools/build_meta.py", line 419, in _build_with_temp_dir
      self.run_setup()
    File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
      exec(code, locals())
    File "<string>", line 929, in <module>
    File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/shutil.py", line 419, in copy
      copyfile(src, dst, follow_symlinks=follow_symlinks)
    File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/shutil.py", line 258, in copyfile
      with open(dst, 'wb') as fdst:
           ^^^^^^^^^^^^^^^
  PermissionError: [Errno 13] Permission denied: '/tmp/casparl/easybuild/build/PyTables/3.9.2/foss-2023b/tables/tables-3.9.2/tables/libblosc2.so'
  error: subprocess-exited-with-error

  × Building wheel for tables (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/bin/python /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /scratch-local/casparl.11613200/eb-pv_73g71/tmp4z0zib8v
  cwd: /tmp/casparl/easybuild/build/PyTables/3.9.2/foss-2023b/tables/tables-3.9.2
  Building wheel for tables (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for tables
Failed to build tables
ERROR: Could not build wheels for tables, which is required to install pyproject.toml-based projects

The error is very misleading, as it complains about a denied permission on /tmp/casparl/easybuild/build/PyTables/3.9.2/foss-2023b/tables/tables-3.9.2/tables/libblosc2.so. The real underlying issue is that PyTables, in it's setup.py, tries to determine if it can find the blosc2 headers, library and runtime library. The difference between the last two is a bit abstract if you ask me, but from what I understand from the setup.py the library is searched while taking e.g. LIBRARY_PATH into account, but the runtime library is searched by trying to do a ctypes.CDLL() on the library name. In EESSI, the library is found (as is stated in the output), but the runtime library is not found, because the ctypes.CDLL() call does not open the library succesfully. To illustrate this:

module load Blosc2/2.13.2-GCCcore-13.2.0
module load Python/3.11.5-GCCcore-13.2.0
$ python -c "import ctypes; ctypes.CDLL('libblosc2.so')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libblosc2.so: cannot open shared object file: No such file or directory

Note that this:

$ LD_LIBRARY_PATH=$LIBRARY_PATH python -c "import ctypes; ctypes.CDLL('libblosc2.so')"
$

Returns succesfully.

Also note that ctypes.cdll.LoadLibrary() has the same issue (and I believe this is actually only a wrapper around ctypes.CDLL).

[casparl@tcn141 PyTables]$ LD_LIBRARY_PATH=$LIBRARY_PATH python -c "import ctypes; ctypes.cdll.LoadLibrary('libblosc2.so')"
[casparl@tcn141 PyTables]$ python -c "import ctypes; ctypes.cdll.LoadLibrary('libblosc2.so')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/ctypes/__init__.py", line 454, in LoadLibrary
    return self._dlltype(name)
           ^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2/software/Python/3.11.5-GCCcore-13.2.0/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libblosc2.so: cannot open shared object file: No such file or directory

We fixed a similar issue raised here in this easyblock PR. That fixed situations where people would call ctypes.CDLL(ctypes.util.find_library('libblosc2')), by making sure that ctypes.util.find_library('libblosc2') returns a full path. However, this doesn't help if a code just calls ctypes.CDLL('libblosc2.so') or ctypes.cdll.LoadLibrary('libblosc2.so) (which many programs also do and is completely valid - the argument doesn't need to be a full path at all, it can just be a library name).

I think we should see if we can apply a similar type of patch so that ctypes.CDLL also works if we pass it a library name, instead of a full path. If we want to be very dirty, we could make sure the call sets (or if it already exists: prepends) LIBRARY_PATH to LD_LIBRARY_PATH, but only for the duration of the call. Or, we analyse if it's a full path, and if not, do a ctypes.util.find_library() call first (for which we had already solved the problem). We should ideally do this again with a change to the Python EasyBlock (as we did for ctypes.util.find_library()), and again make sure the patch is only applied if EasyBuild is configured with RPATH support and filter LD_LIBRARY_PATH.

Both approaches require extensive testing, as it is a pretty fundamental change to a very commonly used module (Python).

@ocaisa: might be of interest to you.

Edited May 08, 2025 by Caspar van Leeuwen