Skip to content

Busyloop while calculating cache keys in `bst build --track`

With BuildStream commit 34ba445f and gnome-modulesets.git commit d9eb944fce94d31f5320cbf6e41e5f29159f3e7b I can cause a busyloop.

First run: bst build --track apps.bst

Then hit CTRL+C at some point.

Eventually all status output will stop, and the main bst progress will peg a CPU at around 100% forever. CTRL+C is ignored in this state, kill -9 seems the only way to stop the process.

Attaching a debugger shows that we are stuck in Element._get_cache_key() for one of the elements that had a track job recently succeed.

Here's a partial, rough Python traceback that I came up with while investigating the issue:

# this is element causing the hang; this method never completes.
Element._cached ()
PullQueue.skip()
list comprehension in queue.py:152
# the elts argument will contain the element causing the hang
Queue.enqueue(self=PullQueue(), elts=[CMakeElement(name='apps/evolution.bst')])
Scheduler.sched()
buildstream._scheduler.Queue.job_done()
# the element argument here will be the element causing the hang
buildstream._scheduler.job.Job.child_complete()
# the 'pid' argument here will correspond to the track process for the element causing the hang
asyncio.unix_events.SafeChildWatcher._do_waitpid()
asyncio.unix_events.SafeChildWatcher._do_waitpid_all()
asyncio.unix_events.BaseChildWatcher._sig_chld()
scheduler.loop.run_forever()
Scheduler.run()
_pipeline.Pipeline.build()

It seems that the issue is: an element tracks successfully, so we try to add it to the pull queue. We check if it's cached in order to find out whether we should pull it, which requires calculating its cache key, which gets stuck in an infinite loop presumably because some of its dependencies are inconsistent.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information