[indexing] Allow setting memory upper limit for indexing jobs to prevent system instability

Problem to Solve

As discussed here we need to set up upper memory limit for the indexing job. Giving access to unlimited memory for gkg may lead to containers getting killed or system becoming unstable due to other processes panics. This is especially important for deployed variant of the knowledge graph.

Proposed Solution

Set up interruption and signalling mechanism if memory usage gets too high during the indexing job. We can run this check during "work stealing" step before each file indexing and later during the analysis stage after each N number of files.