Tools: Implement dynamic dictionary evaluation (ast_evaluator.py)

Summary:

The ASTevaluator class is a wrapper around a Python ASTeval interpreter that evaluates expressions of the form $(...) which are contained in strings nested Python objects (e.g., dictionaries, lists, sets, tuples).

The typical use case for this class is to evaluate expressions contained in the object read in from a YAML file, allowing to embed pieces of Python code inside the YAML expressions to be dynamically evaluated at run-time.

For examples look at the unit test of the ASTevaluator as well as below.

For details of the ASTeval interpreter see: https://docs.python.org/3/library/ast

Syntax:

Examples:

Here are a few snipplets to demonstrate the usage:

Simple Python Session:

>>> from scripts.tools.declarative_def_tools.ast_evaluator import ASTevaluator
>>> ast = ASTevaluator()
>>> dct = {"value": "$(sqrt(2))", "text": "$('%.2f' % value)"}
>>> processed = ast.eval(dct)
>>> processed.dump()
value = 1.4142135623730951
text = '1.41'
>>>

Import with Dynamic Parameters

Assume you have a YAML file as shown here (taken from the unit test):

defaults:
  v01_string: 'test'
  v02_bool:   True
  v03_float:  0.075
  v04_int:    17
  v05_dict:
    v06_selfref: $(v04_int)
    v07_int_div: $(v04_int // 10)
    v08_escape:  $!(blabla % 10)$
    v09_string:  $("a=%.5f" % v03_float)
    v10_forwref: $(v05_dict.v11_math)
    v11_math:    $(sqrt(4))
    v12_backref: $(v05_dict.v11_math)
    v13_string:  $(str("10"))
    v14_list:    [ $(1+2), $(17+4) ]
    v15_list:    $([ 1+2, 17+4 ])
    v16_tuple:   $(( 1, 2 ))
    v17_tuple:   $(( 1+2, "17+4" ))
    v18_string:  $("$(1+2)")
    v19_string:  $!($(1+2)
    v20_float:   $(pi)

This can be evaluated as follows:

>>> from scripts.tools.declarative_def_tools.ast_evaluator import ASTevaluator
>>> spec = yaml.safe_load("path/to/file.yaml")
>>> ast = ASTinterpreter()
>>> params = ast.eval(spec["defaults"], allow_self_ref=True)
>>> params.dump()
v01_string = 'test'
v02_bool = True
v03_float = 0.075
v04_int = 17
v05_dict.v06_selfref = 17
v05_dict.v07_int_div = 1
v05_dict.v08_escape = '$(blabla % 10)$'
v05_dict.v09_string = 'a=0.07500'
v05_dict.v10_forwref = 2.0 # or '$(sqrt(4))' if called with `try_resolve=False`
v05_dict.v11_math = 2.0
v05_dict.v12_backref = 2.0
v05_dict.v13_string = '10'
v05_dict.v14_list = [3, 21]
v05_dict.v15_list = [3, 21]
v05_dict.v16_tuple = (1, 2)
v05_dict.v17_tuple = (3, '17+4')
v05_dict.v18_string = '$(1+2)'
v05_dict.v19_string = '$(3'
v05_dict.v20_float = 3.141592653589793
>>>

By the AST evaluation call the dictionary params will get evaluated. The resulting values are denoted in the YAML code above for information.