[Skims & SBOM] Adjust job architecture
Problem to solve
Currently, machine and SBOM jobs are defined an run using integrates/jobs/
This approach has several drawbacks:
- It does not have ANY tests, so very frequently after a change, it breaks machine executions.
- As we progress towards a more optimized way to run machine and sbom, we find the need to send more and more specific information for the setup of the job (e.g. checks, specific paths, environment info), this is hard to manage only using the
adittional_info
attribute of the batch item, since it isn't typed and we may find ourselved hitting the size limit soon. - It relays on the API to get information related to the group (language, files and environments) which have seen some performance problems.
Intended users
Machine and SBOM users
Proposal
Instead of depending on this fragile architecture, the following procedure is proposed:
- Create the YAML file needed for machine and SBOM executions right when the
queue_new_job
function is called, and upload it to the bucket. - Get download links for environment files and add the links to batch item.
- Add link information to download the yaml file in the batch item directly
- In
integrates/jobs/execute_machine/entrypoint.sh
the procedure would be: download yaml file, download apk files, clone the repo, run machine, submit task to process results.
Test plan
Add test to create yaml files in different circumstances (reattacks, full executions, updating a yaml file if the job has not run yet)
Steps
-
Let's experiment this architecture first for SBOM jobs, since right now it's a less critical flow. -
If it works, use the same proposal for machine jobs. -
Tests, tests, tests and more tests.
What does success look like, and how can we measure that?
A more resilient machine architecture, that allows us to customize and fully test the configuration file used for the executions and ensures we have no more big break downs.
Edited by Fabio Lagos