MVP of custom workflows

Description

Fixes #441 (closed)

This merge request introduces a feature where a user can optionally paste a repository URL for a workflow when launching an instance. The information for launching the workflow is stored in the metadata for the instance, and injected into the boot script.

During the initial instance build phase our instance provisioning code (powered by Ansible) installs the repo2docker tool. Next it runs repo2docker to pull down the user-specified workflow from the specified repository, and builds a container image from the repository. Then we use docker to start container from the image. We start the container in daemon mode so that it will be started automatically after reboot/unshelving/shutdown. We expose a port on the instance for accessing the data science workbench service (8888) when invoking docker. Once the container is started successfully we grab the authentication token from the Docker logs and write it to /dev/console. There is also a scheduled (cron) job which will do this regularly (every minute at the moment - to be optimized later) so that when the authentication token changes (like after a reboot) the latest authentication token will always be written to the console.

The Exosphere client application regularly check for, and then grabs the authentica token from the console log (via the OpenStack API) and then considers the workflow container active. The workflow button on the server details page then becomes active.

When the user presses the workflow button it launches a special URL in a new browser tab. This URL points to the configured UAP (User Application Proxy) and includes the IP address of the instance, the exposed port, and the authentication token. This results in a TLS encrypted connection to the running, pre-authenticated data science workbench. (Note: There is no need for the user to locate and manually enter an authentication token into Jupyter.)

This MR addresses the following success criteria for #441 (closed):

• Easy to create a reproducible workflow (including specifying the environment and dependencies)

With Reproducible Execution Environment Specification (or REES): "Any directory containing zero or more files from the Configuration Files list is a valid reproducible execution environment as defined by the REES."

• Leverages community standards and best-practices for reproducible computational standards

Like Binder we use Reproducible Execution Environment Specification (or REES) to "automate and encourage existing community best practices for reproducible computational environments. This includes installing packages using community-standard specification files and their corresponding tools"

• A researcher can develop a workflow on their local machine

See REES above. As long as the root directory of the workflow source code complies with REES then they can develop their workflow anywhere.

• Easy to publish a reproducible workflow

We leverage the open source repo2docker tool from the Jupyter project to pull down a REES-compliant workflow from common public source repositories where researchers publish workflows (only Git repositories in this MR, will add other repositories with DOIs later), and turn it into a fully encapsulated, executable workflow environment.

• Can run the same workflow on most common computational platforms which researchers around the world have access to
  • For example their own laptop, server, HPC, research cloud, commercial cloud, etc.
  • In other words not locked into the Exosphere UI, OpenStack, Kubernetes, free public services like mybinder.org, or closed commercial services like Code Ocean, Github Codespaces, etc.
• Launch a workflow as easily as on mybinder.org, but on hardware which is powerful and/or specialized (GPUs)
• Ability to run a workflow on publically funded research clouds like Jetstream, Nectar, CyVerse, DOE labs, NCSA, Chameleon, etc.
• Workflows can run indefinitely, unlike on mybinder.org (only limited by cloud provider quotas - if applicable)

We make it easy to run these workflows (no knowledge of command-line or container technology required) on any OpenStack cloud (including public research clouds with powerful hardware).

A workflow started on OpenStack can run for as long as the user desires (limited only by their quota).

At the same time a researcher can use the repo2docker tool we use to run the same workflow on most common computational platforms (laptop, server, HPC, research cloud, commercial cloud, etc). It will require them to install a container runtime, and some familiarity with the command-line, but this means they are not locked in to any particular platform.

• No ongoing maintenance burden on publishers of workflows for security updates, etc. (as opposed to long-lived virtual machine images)
• No ongoing maintenance burden on research cloud operators who provide a curated community catalog of workflows, but who are also responsible for the security of their infrastructure (as opposed to long-lived virtual machine images)

Workflows are created every time from a repository specification, and run in an isolated container environment. For most workflows the latest, most secure version of the base operating system will be used every time the workflow is instantiated.

The exception is workflows which use Dockerfiles to explicitly lock down the base operating system to a specific version of the base operating system - an advanced usage. See Use a Dockerfile for your Binder repository for more information.

We rely on the security model of the Docker container runtime to run the workflows in a fully encapsulated environment.

Which features may we decide to address later? (depending on community demand)

• Possible to share a workflow privately (pre-publication)

Could be accomplished by adding ability to enter credentials for private repositories

Not mentioned in issue #441 (closed), and out of scope for this merge request:

• Data management
• Browsable, community-curated catalogs of workflows

How to Test

• Modify config.js to have the following values for instanceConfigMgtRepoUrl and instanceConfigMgtRepoCheckout:

  instanceConfigMgtRepoUrl: "https://gitlab.com/julianpistorius/exosphere.git",
  instanceConfigMgtRepoCheckout: "issue-441-custom-workflows"

• Reload the Exosphere application
• Start process of creating a new server
  • Note: I only tested with JS-API-Featured-Ubuntu20-Latest and JS-API-Featured-CentOS8-Latest images, m1.small flavor, on Jetstream IU provider
• Expand Advanced Options
• Under Boot Script there should be a section called Build and launch a workflow
• Press on Show
• There should be a note with an orange background that says: Note: Workflows is an experimental feature
• Enter an invalid URL into the Git repository URL input field: github.com/binder-examples/minimal-dockerfile
• Ensure that the Remove workflow and Create buttons are disabled
• Add https:// in front of the URL so that it is: https://github.com/binder-examples/minimal-dockerfile
  • Note: This is a good repo for testing because it has a very fast build time
  • We should also test other REES-compatible repos
• The Remove workflow and Create buttons should be enabled
• Press the Remove workflow button
• Ensure that the Git repository URL input field is cleared
• Paste a valid repository URL into the Git repository URL input field: https://github.com/binder-examples/minimal-dockerfile
• Press the server Create button
• Press on the name of the server you created to go to the server details view
• Scroll down to the Interactions section
• Ensure that you see a button with text Workflow
• The Workflow button should be disabled
• Press the information tooltip button to the right of the Workflow button
• Ensure that you see the following text in the tooltip window: Instance is still booting or the workflow is still deploying, check back in a few minutes (Known issue: Currently this message is partially obscured by the side menu.)
• Ensure that after a few minutes the Workflow button becomes enabled (with a green, rounded square before it)
• Press the information tooltip button to the right of the Workflow button
• Ensure that you see the following text in the tooltip window:

  Status: Ready
  Description: Access the workflow launched with this instance

• Press on the Workflow button
• Ensure that a new tab opens in the browser and that this browser tab contains a Jupyter interface
• Ensure that you are logged in to the Jupyter notebook service, and that it does not prompt you for a token or password

Screen recording

Screenshots

Post-Merge Checklist

• Create follow-up issues for anything left over for another merge request
  • If any technical debt is introduced, assign these follow-up issues to the MR submitter