Skip to content

Use generator expressions to reduce mem usages

Tan Le requested to merge fix-generator-expressions into main

We do not need to have a full list created in memory. Instead, they only need to iterate over the elements one at a time.

Given this code

sum([info.length_tokens for info in self.content])

It will build a full list of length tokens in memory, iterate over those values, and, when the reference is no longer needed, delete the list.

With generator, this operation does not store the full sequence of numbers in memory. In fact, only two numbers need be stored during any given iteration of the sum: the current value of the sum, and the number being added to it.

https://peps.python.org/pep-0289/

Edited by Tan Le

Merge request reports