Browser-accepted TLS for connections to interactive services on users' cloud instances
Problem/Opportunity Statement
The web browser is the universal UI platform. Exosphere is both served to and used in the browser. Most of the instance interactions that Exosphere exposes (Guacamole, JupyterLab) also run in the browser. These interactions are ultimately served by cloud instances that are set up without human intervention or special credentials/access. We generally need the ability to create many cloud instances and have them serve things that users can access in their web browser.
To avoid compromising network security, web browsers are picky about the sort of network connections they will make without hassling or scaring the user. Our choices are:
- HTTPS (TLS) to a public hostname where the server bears a certificate signed by one of a set of browser-accepted authorities (e.g. Let's Encrypt).
- Connection to a server running on the user's own computer (localhost)
Achieving (2) requires additional software on the user's local computer. This could be feasible for more computationally-savvy users on a desktop platform, but not for less-computationally-savvy users, or those on a mobile operating system. So, we cannot rely on it. That leaves us with (1). Instance interactions need to be served with browser-accepted TLS.
Serving anything with browser-accepted TLS requires both:
- A hostname on a publicly registered domain. It's easy to generate many hostnames for a registered domain, but domain registration generally requires money and human intervention, which matters for the next point.
- Cooperation of a certificate authority. Not many years ago, this required money for each new certificate. It doesn't anymore, but CAs still either impose rate limits per registered domain (Let's Encrypt), or they require prior human account registration (ZeroSSL). So, it's still hard to do automatically, for untrusted parties, without human intervention.
We currently satisfy these requirements with User Application Proxy (UAP) servers. There's a UAP deployed at each OpenStack cloud (or region of a cloud) whose operators want Exosphere's instance interactions to work. Its job is to proxy and secure network connections to interactive services that run on cloud instances that are deployed with Exosphere. On the upstream end (toward the instance), the UAP only makes connections to instances on local Neutron network segments. On the downstream end (toward the user's browser), it serves these connections with a wildcard hostname and TLS certificate. Putting a few cloud-specific UAPs between users and their instance interactions means we only have a few hostnames and certificates to manage, so it's feasible for the Exosphere developers or cloud operators to manage them.
The current UAP arrangement does not work for connections to instances that:
- Are created on OpenStack clouds that don't have a UAP server, because the cloud's operators haven't set one up (or are unwilling to)
- Don't have public (typically floating) IP addresses on the same Neutron network segment as the cloud's UAP server.
- Public IPv4 addresses are increasingly scarce and
- Require rock-solid, un-MitM-able upstream TLS connections to the UAP. Currently, the upstream HTTP connections between UAP and the instance rely on the underlying cloud's (Neutron's) network segment encapsulation, hence the current restriction that a UAP can only proxy connections for instances on the same Neutron network. This both isn't the most secure, and it doesn't scale very well.
- Probably other situations too.
What would success / a fix look like?
Perhaps, some kind of proxy arrangement that:
- Doesn't require instances to all have public IPs
- Doesn't require the proxy server to be located on the same Neutron network segment as the instances, so that Exosphere project (or anyone else) could host a central one for people accessing multiple clouds to use
- Could be run by a cloud operator (or locally by a computationally-savvy user) who doesn't want to trust a third party's proxy server
We can generate a token/secret that is shared between Exosphere client and instance at the instance creation time. So, we have some basis for mutual authentication.
Possible technologies and techniques which could help us achieve this:
- StackHPC Zenith achieves this with instances creating long-lived SSH connections (with TCP port forwarding) from the instance to the Zenith server.
- A VPN technology like Tailscale but not proprietary. Technologies in this space include Wireguard and Hyprspace
- Something like Hashicorp Boundary