Hub Integration

The Hub will be a central point where all instances can create, read, update, and delete Tasks. It will be helpful for sharing Tasks and quickly getting started with Polyapp.

Considered Designs

Option 1: Import and Export

Docker's Hub works by allowing people to download Dockerfiles. When you want to upload your own Dockerfile, you need to log in to Docker and then perform a download.

Pros

  • Auth is easy: Just use people's Google Accounts.
  • Involves developing the Import / Export functionality which I need for other reasons too

Cons

  • Long customer workflow. Optimistically, steps might include: (a) Click a button in Polyapp which downloads a file (b) Navigate to the Hub (c) Log in redirect (d) Upload the downloaded file (e) Press Done to finish adding your file (f) View that the file was successfully uploaded and examine the entry in Hub
  • Will the Hub need to be coded in something other than Polyapp to ensure the website performs well and is secure? What happens if someone wants to use a company-internal Hub instead of a public Hub? Will this require a substantial amount of development time?

Option 2: Server - Server Communication

Many applications support public APIs which let you read data from a system. For instance, there is an Open Maps API, or a paid Google Maps API. Some of these systems also allow you to create, update, and delete resources in the remote system, with restrictions of course. Polyapp could implement instance to instance communication using an API Key for Auth.

Pros

  • Superior user experience: check a box and the Task will automatically save to the remote Hub. To view the remote Hub, we could create a Task in Polyapp like the Home Task which loads in data from a remote source when the Task loads.
  • Uses Polyapp as both sides of the server. So it should be easy to create this functionality and not require mixing in other types of development like traditional web development.
  • Uses Polyapp as both sides of the server, so if someone wants to use an internal Hub they just point their Polyapp at a different URL.
  • When Polyapp is enhanced with new functionality, I can try to reuse that functionality elsewhere. Perhaps Polyapp's API functionality could be expanded later to allow for more types of server - server communication with a different set of permissions in the system.

Cons

  • Auth is error-prone: API Keys must be CRUD-able, and there would also have to be an ID issued to allow a key - value lookup since an instance does not naturally have a unique ID issued by the Hub server when it starts up.
  • API Keys will be relatively expensive. The correct way to store such keys in Google Cloud is with their Secrets Manager. Unfortunately the Secrets Manager costs $0.06 / version / month. For a client there might be charges up to $0.50 / month. This is easily manageable. For a hub server the charges would be up to $0.50 / month per client. The charge cost could be reduced to $0.06 / month in both cases by keeping all information except the API Key outside of the Secrets Manager and deleting old API Keys.

Design

When choosing an option my first priority is to choose the one which yields the best theoretical user experience. In this case, that is Option 2. My second priority is to consider which option leads to Polyapp having a more robust feature set. That is also Option 2. Therefore I went with Option 2.

Existing Functionality Changes

Changes to Tasks

  • The "Edit Task" Task will need a new checkbox which enables sharing the Task.

Bot / Action / Code Changes

  • "Edit Task" will need to gather all pertinent information about the Task (all Datas used to design it, Tasks, Schemas, UXes, Bots and Actions) and package it together in a request to the remote server. The request will need to be made and the response handled.

New Functionality

Tasks

  • The "Hub" needs a Task which holds the information we send over.
  • The "client" (the sending server) may need to store some information mapping things from the current environment into the destination environment.
  • The "Hub" may need to store some information mapping data coming from the external source into information in the internal source.

Bot / Action / Code

None for those Tasks.

Other Database

  • A new User and Role document which grants access to a subset of all Polyapp functionality. The incoming requests will be granted some access based on these documents while their requests are in flight; additional access control will also be needed.

New APIs

  • /APICredentials will issue the receiving server a new set of API Credentials including a new API ID, API Key, and data about which specific industry/domain/schema it can access when a GET command is issued.
  • /APICredentials will support a DELETE
  • /api/t/ APIs. Currently /t/ uses POST to update the server and /t/ uses GET to read from the server. We need to expose this functionality to the API for a limited set of industry/domain/schema.
  • /api/t/ PUT API. The current functionality only creates a new document when the "Create Data" button is pressed on the "Change Data" page and some custom code runs which creates a new Document. It is also possible for a custom Bot / Action / code to create a new Data. In both cases, you must already be using a Polyapp Task in order to create a new Data. This does not make sense for a remote server, which needs to PUT data without having to make a GETDataTables request and then do a POST. By creating a new PUT API we can "create" document on the remote server.
  • /api/t/ DELETE API. There was no DELETE API for the same reason there was no PUT API but we need one for the same reasons we need a PUT API.

New Middleware

  • APIAuthenticate and APIAuthorize. Authenticate will ensure the provided API ID and API Key match what we have in Secrets. Authorize will make sure the request is only trying to PUT or POST to the Task / Schema / UX created by the remote server.

New Tests

In the past I haven't included tests for the primary APIs because I had to verify their functionality via the UI. My thinking was: "When I originally created Polyapp I tried to add tests, but then I deleted all of that code. When I created version 2, I also added some tests, but then I deleted all of that code. Why slow myself down with UI-based testing this time?" I am now certain I will not throw out this version so it makes sense to start to add some end to end tests (there are already some unit tests and integration tests for package allDB).

  • Add testing for creating and deleting API Credentials.
  • Add testing for PUT (create), POST (update), GET (read), GETDataTable (read many), DELETE.
  • Add testing for the middleware.

Implementation Details

  • This functionality will be implemented in the master branch because I am the only one using Polyapp right now (as far as I know).
  • I will need to improve the open source CI/CD development process and get it ready for multiple environments. After this functionality is implemented, I anticipate needing to support more than one deployment environment since I will need to support a production-grade "Hub" environment and a development environment.