Long-running tasks can show no output for a very long time
Summary
There are places in buildstream's start-up where long-running tasks are performed, but no indication of progress is given, making it hard to distinguish a long-running task from a task that is stuck.
For example, we have the "Loading elements" and "Resolving elements" timed activities, where there can be a very long time between them starting and them finishing for a sufficiently large set of elements.
Steps to reproduce
Find a sufficiently-large set of elements - https://gitlab.com/jennis/debian-stretch-bst is good for this purpose.
Load the pipeline for that set with bst show.
What is the current bug behavior?
You will see a very long time is spent in "Loading elements" with no other output.
What is the expected correct behavior?
Some kind of progress indicator would be preferred. A simple "N elements loaded" would be sufficient, but an "N out of M elements loaded" would be nicer.
Possible fixes
For "loading elements" in particular, the timed_activity occurs in buildstream/_project.py:452 (in load_elements), but an element finishes loading in buildstream/_loader/loader.py:442 (in _collect_element). The length of a Loader's _meta_elements is not a useful count of how many elements have been loaded because multiple loaders will exist if junctions are used.
"Resolving elements" has the timed_activity in buildstream/_project.py:457, but an element finishes being resolved in buildstream/element.py:1004 (in _new_from_meta). Unfortunately, there is no list for every resolved element. The closest that exists is that each Element class has a dict of __instantiated_elements.
Since there are no good places for a list of current/total elements, the best ways to count progress would be:
- have a counter in the Context
- have a Progress object that gets passed down the call stack
Since printing output every time an element is loaded or resolved would consume a lot of resources for marginal benefit, a threshold should be specified for when to print (e.g. every N items, or every N seconds)