Devours memory and stack traces when recursive variables defined
Summary
I accidentally created a recursively defined variable in this commit [1]. Attempting to build locally gave me the following stack trace after ~15 minutes of making my machine unusable:
Traceback (most recent call last):
File "/home/thomascoldrick/.local/bin/bst", line 8, in <module>
sys.exit(cli())
File "/home/thomascoldrick/.local/lib/python3.5/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/thomascoldrick/buildstream/buildstream/_frontend/cli.py", line 162, in override_main
standalone_mode=standalone_mode, **extra)
File "/home/thomascoldrick/.local/lib/python3.5/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/thomascoldrick/.local/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/thomascoldrick/.local/lib/python3.5/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/thomascoldrick/.local/lib/python3.5/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/thomascoldrick/.local/lib/python3.5/site-packages/click/decorators.py", line 27, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/home/thomascoldrick/buildstream/buildstream/_frontend/cli.py", line 330, in build
build_all=all_)
File "/home/thomascoldrick/buildstream/buildstream/_stream.py", line 184, in build
dynamic_plan=True)
File "/home/thomascoldrick/buildstream/buildstream/_stream.py", line 871, in _load
fetch_subprojects=fetch_subprojects)
File "/home/thomascoldrick/buildstream/buildstream/_pipeline.py", line 111, in load
fetch_subprojects=fetch_subprojects)
File "/home/thomascoldrick/buildstream/buildstream/_project.py", line 321, in load_elements
for meta in meta_elements
File "/home/thomascoldrick/buildstream/buildstream/_project.py", line 321, in <listcomp>
for meta in meta_elements
File "/home/thomascoldrick/buildstream/buildstream/element.py", line 926, in _new_from_meta
dependency = Element._new_from_meta(meta_dep, artifacts)
File "/home/thomascoldrick/buildstream/buildstream/element.py", line 929, in _new_from_meta
dependency = Element._new_from_meta(meta_dep, artifacts)
File "/home/thomascoldrick/buildstream/buildstream/element.py", line 909, in _new_from_meta
element = meta.project.create_element(artifacts, meta, first_pass=meta.first_pass)
File "/home/thomascoldrick/buildstream/buildstream/_project.py", line 232, in create_element
return self.config.element_factory.create(self._context, self, artifacts, meta)
File "/home/thomascoldrick/buildstream/buildstream/_elementfactory.py", line 61, in create
element = element_type(context, project, artifacts, meta, default_config)
File "/home/thomascoldrick/buildstream/buildstream/element.py", line 248, in __init__
self.__variables = Variables(variables)
File "/home/thomascoldrick/buildstream/buildstream/_variables.py", line 48, in __init__
self.variables = self._resolve(node)
File "/home/thomascoldrick/buildstream/buildstream/_variables.py", line 148, in _resolve
resolved, unmatched = resolve_one(resolved)
File "/home/thomascoldrick/buildstream/buildstream/_variables.py", line 134, in resolve_one
resolved_var, item_unmatched = self._subst(value, variables)
File "/home/thomascoldrick/buildstream/buildstream/_variables.py", line 102, in _subst
replacement = re.sub(_VARIABLE_MATCH, subst_callback, string)
File "/usr/lib/python3.5/re.py", line 182, in sub
return _compile(pattern, flags).sub(repl, string, count)
MemoryError However on the CI things failed neatly [2] despite both using 1.1.6
[1] freedesktop-sdk/freedesktop-sdk@9c53d6a5 [2] https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/90954727
Steps to reproduce
I was reproducing this bug reliably on my machine by running make runtime export on the commit linked. However this may not be reproduced on every machine.
What is the current bug behavior?
Buildstream steals all the memory and crashes, sometimes, when recursive variables are defined.
What is the expected correct behavior?
The nice error message which I got in the CI
Possible fixes
Potentially a check for recursion in variables while they're being resolved?
Other relevant information
- BuildStream version affected: 1.1.6
Edited by toscalix