utils.safe_copy() fails to copy *to* a location where leading directories do not exist

Summary

When writing a source plugin, I tried to copy files (on disk) to a similar location in BuildStream using utils.safe_copy('opt/baz/bin/foo', os.path.join(directory, 'opt/baz/bin/foo'))

Now, this function fails because the directories opt/baz/bin do not exist. Note, we're utimately calling shutil.copy_file() here: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/utils.py#L266

A simple fix for this is to execute:

os.makedirs(os.path.dirname(os.path.join(directory, 'opt/baz/bin/foo')), exist_ok=True)

before we call utils.safe_copy, within the plugin.

It seems that this fix could also be implemented within the utils.safe_copy() function itself. However, @juergbi mentioned that we should not expand the scope of safe_copy() as, strictly speaking, it would even be an API break

Perhaps, we could consider an additional helper method?

I've opened this issue for discussion

changed the description

Have you seen _copy_directories in utils.py? It may do what you need, although your makedirs snippet looks equally correct to me.

Issue #476 (closed) may be related - I'm trying to write a document that explains exactly how files get imported at the moment.

Ahh interesting, however this looks like it's intended to be API private. It's only used within utils.py itself.

Would modifying the safe_copy() function to accept a default create_dirs=False argument, or similar, be an option? Or is this still classed as a break of API?

added Important To Do labels

changed milestone to %BuildStream_v1.4

These functions should be private anyway in BuildStream 2, and I don't think they should implicitly create leading directories either (it's probably valuable to have them fail instead when leading directories don't exist, as it could indicate a problem).

closed