Hang on "Initializing remote caches": grpcio hang on Fedora 33

Summary

$ bst build core.bst sdk.bst
[--:--:--][][] STATUS  Cache usage recomputed: 10B / infinity (0%)
[--:--:--][][] START   Build
[--:--:--][][] START   Loading elements
[00:00:04][][] SUCCESS Loading elements
[--:--:--][][] START   Resolving elements
[00:00:02][][] SUCCESS Resolving elements
[--:--:--][][] START   Initializing remote caches

With buildstream-1.6.0-1.fc33, bst-external-0.21.0-1.fc33, and python3-3.9.0~rc2-1.fc33, this last step never completes.

Steps to reproduce

Clone gnome-build-meta, then try to bst build core.bst sdk.bst.

What is the current bug behavior?

Hang before anything starts to build.

What is the expected correct behavior?

It should build!

Relevant logs and/or screenshots

@abderrahimk asked for some strace. Here is the very end before it hangs performing a read():

openat(AT_FDCWD, "/dev/shm/mF8sKU", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
write(5, "\1\0\0\0\0\0\0\0\200\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 32) = 32
mmap(NULL, 32, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = 0x7efc81be5000
link("/dev/shm/mF8sKU", "/dev/shm/sem.mp-e3jut377") = 0
fstat(5, {st_mode=S_IFREG|0600, st_size=32, ...}) = 0
unlink("/dev/shm/mF8sKU")               = 0
close(5)                                = 0
unlink("/dev/shm/sem.mp-e3jut377")      = 0
getpid()                                = 55091
getpid()                                = 55091
lstat("/dev/shm/7A80eV", 0x7ffd2832ba00) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/shm/7A80eV", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
write(5, "\377\377\377\177\0\0\0\0\200\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 32) = 32
mmap(NULL, 32, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = 0x7efc81be4000
link("/dev/shm/7A80eV", "/dev/shm/sem.mp-7auqask5") = 0
fstat(5, {st_mode=S_IFREG|0600, st_size=32, ...}) = 0
unlink("/dev/shm/7A80eV")               = 0
close(5)                                = 0
unlink("/dev/shm/sem.mp-7auqask5")      = 0
getpid()                                = 55091
rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
getpid()                                = 55091
stat("/usr/lib64/python3.9/multiprocessing", {st_mode=S_IFDIR|0755, st_size=576, ...}) = 0
stat("/usr/lib64/python3.9/multiprocessing/popen_fork.py", {st_mode=S_IFREG|0644, st_size=2377, ...}) = 0
stat("/usr/lib64/python3.9/multiprocessing/popen_fork.py", {st_mode=S_IFREG|0644, st_size=2377, ...}) = 0
openat(AT_FDCWD, "/usr/lib64/python3.9/multiprocessing/__pycache__/popen_fork.cpython-39.pyc", O_RDONLY|O_CLOEXEC) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0
ioctl(5, TCGETS, 0x7ffd2832b4a0)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(5, 0, SEEK_CUR)                   = 0
lseek(5, 0, SEEK_CUR)                   = 0
fstat(5, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0
read(5, "a\r\r\n\0\0\0\0A\202b_I\t\0\0\343\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2513) = 2512
read(5, "", 1)                          = 0
close(5)                                = 0
pipe2([5, 6], O_CLOEXEC)                = 0
pipe2([7, 8], O_CLOEXEC)                = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7efc81549a10) = 55632
close(6)                                = 0
close(7)                                = 0
getpid()                                = 55091
rt_sigprocmask(SIG_SETMASK, [], [INT], 8) = 0
read(3,

Possible fixes

@abderrahimk suggests downgrading to python 3.8 might fix this. I'm about to try running buildstream in a F32 container to see.

Other relevant information

Edited by Seppo Yli-Olli