Can we replace/simplify some os.path.join()s for performance?

Background

This issue was part of a list of things to investigate regarding performance. Previously described by @samthursfield as:

As shown in the profile attached to !247 (merged), we spend a lot of time in os.path.join() -- for example 8 seconds of a 50 second staging operation. This would be worth optimizing if possible.

os.path.join contains some logic in order to cover many cases. It is possible that when used in a tight loop (while iterating over lots of files) it might be worth investigating. Especially if we are sure that the join we expect a is simple string concatenation.

We can consider extending this for other costly os.path functions.

Task description (in order)

  • Figure out which os.path.join() calls are indeed in a tight loop(s)
  • Determine if those calls are simple enough & replecable
  • Do performance comparisons with various projects
  • Have a consensus on which parts to replace (consider readability & long term roadmap)

Acceptance Criteria

It should provide significant performance gain while not complicating the codebase and making it unreadable.

Edited by Tristan Maat