Analyzer timeout enforcement

Description

The duration of some SAST failed jobs can run an order of magnitude or two longer than the failed job duration average. We could implement a timer and kill signal in the analyzers which would prevent the large spikes in failed jobs (see job timings in SAST retro).

Solution approaches

1. Introduce timeout-enforced context.Context to CLI operations at the command module level

   Pros:

   • Relatively less eng effort (compared to approach 2).
   • Applies the same logic to all the analyzers at the root level.

   Cons:

   • Cannot guarantee graceful exits for nested operations which are dependent on external system resources as it would cause leaks.
   • Need to introduce and manage a new CI variable to override the default timeout duration. We can utilize the existing timeout CI variable.

2. Introduce timeout-enforced context.Context inside each analyzer level [Recommended]

   Pros:

   • Can guarantee graceful exits for nested operations which are dependent on external system resources.

   Cons:

   • Need to introduce and manage a new CI variable to override the default timeout duration. We can utilize the existing timeout CI variable.
   • More engineering effort. Better solutions are available for SAST analyzers(see next approach).

3. Use the timeout CI variable for all SAST jobs

   Pros:

   • No need to introduce a new CI variable to manage timeouts.
   • No additional effort or code changes are required as this CI variable suits best for SAST analyzers which are not dependent on external system resources.

   Cons:

   • Cannot guarantee graceful exits for nested operations which are dependent on external system resources as it would cause leaks.