Skip to content

add warnings for unreasonable input sizes

Miron van der Kolk requested to merge fix/631-parsing-recursion-error into develop

Fixes: #631 (closed)

Adds warnings for parsing unreasonably large katscript (both in total size and maximum line size)

It is not easy to support arbitrarily large input lines, since the parses uses recursion by design and python's recursion limit is quite conservative.

Finesse supports infinitely sized katscript in principle, in practice you are rather limited by compute and memory limits: parsing 200kb of just mirror elements takes 40 seconds and 8 gb of memory on my machine. This might actually not be a limit of the parser, but the builder creating the python structures for all these mirrors.

Line length warnings

Input

import finesse

phases = 600

model = finesse.Model()
model.parse("m m1 R=0.9 T=0.1")
model.parse(f"sweep(m1.phi, {[i for i in range(phases)]}, True)")

Output

/home/miron/src/finesse3/src/finesse/script/parser.py:142: KatParsingInputSizeWarning: Unreasonable line length: 602 words in line 0.
Low performance or recursion error possible.
Content: 'sweep(m1.phi, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,...'
  warnings.warn(msg, KatParsingInputSizeWarning, stacklevel=1)

Making to long of a line (setting phases to 1200 on my machine)

/home/miron/src/finesse3/src/finesse/script/parser.py:142: KatParsingInputSizeWarning: Unreasonable line length: 1202 words in line 0.
Low performance or recursion error possible.
Content: 'sweep(m1.phi, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,...'
  warnings.warn(msg, KatParsingInputSizeWarning, stacklevel=1)
Traceback (most recent call last):
  File "/home/miron/src/finesse3/test.py", line 7, in <module>
    model.parse(f"sweep(m1.phi, {[i for i in range(phases)]}, True)")
  File "/home/miron/src/finesse3/src/finesse/model.py", line 78, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miron/src/finesse3/src/finesse/model.py", line 2229, in parse
    parse(text, model=self, spec=spec)
  File "/home/miron/src/finesse3/src/finesse/script/__init__.py", line 44, in parse
    return compiler.compile(text, model=model)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miron/src/finesse3/src/finesse/script/compiler.py", line 154, in compile
    return self.compile_file(StringIO(string), **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miron/src/finesse3/src/finesse/script/compiler.py", line 197, in compile_file
    script = self._parser.parse_file(fobj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miron/src/finesse3/src/finesse/script/parser.py", line 120, in parse_file
    raise KatParsingError(msg)
finesse.script.exceptions.KatParsingError: 
Recursion limit reached during parsing. Probably caused by an unreasonable line length. 
Try increasing recursion limit with `sys.setrecursionlimit` or use the python interface.

Total file size

input

import finesse

n_lines = 1000

# 1000 lines is twice the size of the finesse ligo model (500 lines ~10kb)

model = finesse.Model()
model.parse("\n".join(f"m m{i} R=0.99 T=0.01" for i in range(n_lines)))

output

/home/miron/src/finesse3/src/finesse/script/parser.py:149: KatParsingInputSizeWarning: Unreasonable script size: 20 kb. Low performance or recursion error possible.
  warnings.warn(msg, KatParsingInputSizeWarning, stacklevel=1)
Edited by Miron van der Kolk

Merge request reports