utils._ensure_real_directory() very slow in some cases
I have been experiencing slow staging performance. As a data point, bst shell base.bst
inside the gnome-modulesets.git project takes nearly 2 minutes to actually open a shell prompt.
I profiled the operation with Pyflame and discovered a lot of time is spent during staging in utils.link_files()
(26 seconds) > utils._ensure_real_directory()
> os.path.realpath()
(~19 seconds).
Looking at the code for os.path.realpath(), it seems like the issue is just that the Python standard library implementation of that function is very inefficient.
Removing the os.path.realpath()
calls from utils._ensure_real_directory()
saves around 20 seconds of runtime; we shouldn't remove this check, but we should definitely look for how to speed it up. One option would be to try importing the realpath()
from the system's C library using the ctypes module and calling that instead.