Model tuning: Mix & match Jupyter and HPC-based tuning runs?

There are two options:

• Mix the outputs of Jupyter-based and HPC-based training runs in the same set of folder tree

• Separate the outputs of Jupyter-based and HPC-based training runs in two folder trees

Mixing the outputs

Rationale:

• (+) It doesn't matter which mechanism is being used to train a network (Jupyter or batch)
• (-) It can confuse new learners

Separating the outputs

• (+) Less confusion for new learners
• (+) Fewer risks of overwriting results
• (-) Will have to do extra work if we want to combine the results (e.g. by creating the third tree that has symlinks to the desired outputs)

DECISION 2025: We currently opt to separate the outputs.