Rebuild base environment creation / selection
Problem to solve
As a user I want to be able to easily customize and create my own base environment. Managing dependencies and interdependencies with a pre-fixed base environment (as we have it today) is tideous, as you need to define your requirements, publish, wait until it fails, adapt the requirements again and start all over. This might take a few hours to up to days until the setup works (or it might not even work).
To eliminate this, we should enable easy creation and customization of base environments before publishing, where the image is then "fixed" to a code repository. This, of course, can be adapted at any point in time.
As discussed with @erika.torres, we see currently three, not excluding, main options:
- allow (and offer) open environment.yaml file template where the user can write and specify environemnt configurations.
- Create a environment creator (wizard), where users can select a number of standard libraries via dropdown. E.g. python version (1,2,3.4,3,5, etc). This should also check for interdependencies / compatibility with other additional libraries (e.g. tensorflow v. 2).
- Import a docker image via dockerhub.
The following pros / cons exist for the above:
- Being open it allows for deep customization but might open up secrets / vulnerabilities that MLReef has.
- Easy to use and probably easy to implement, but might still be limiting to the list of available libraries we specify.
- Easy option but requires external (dockerhub) services to work properly. Also need to review how this works with "Nautilus" (with and without internet). This solution might be an additional featuer to 1 or 2.
Question: What concequences could this have to our dind architecture? (compatibility, performance, permanent storage)