Resolve "Make it easier to run production version on local machine"

Closes #304 (closed)

The current state of this MR is that there are now two Baserow "environments" in this repo and the old demo environment has been deleted. This MR description below will be incorporated into baserows doc's once everything has been finalized.

The Local Baserow Environment

$ git clone https://gitlab.com/bramw/baserow.git 
$ cd baserow
$ docker-compose up 
$ # everything will be built, started, migrated and started in the non dev local mode ready for immediate usage.

This will run the local version of baserow (gunicorn, prod nuxt etc) with everything being automatically setup. My reasoning is:

1. This is the simplest and most standard docker-compose workflow. As a result to have the simplest and quickest new user checkout / install / play around with baserow locally experience I wanted docker-compose up to give you the more secure and faster version of baserow.
2. Also because docker-compose up on its own is the most intuitive or obvious first command someone might try when checking out baserow after seeing compose files I wanted these users to also get the fastest and most secure experience.
3. As I believe a much smaller minority of people will want to checkout and dive right into a dev workflow, so I made the dev version the "non-standard" you have to use a different compose file workflow. Additionally this makes the dev version more opt-in, which i prefer as it is the slower and less secure version. Hence by being opt-in the user has to make the decision, yes, i want the dev version, and not accidentally get it by default.

The Dev Baserow Environment

$ git clone https://gitlab.com/bramw/baserow.git 
$ cd baserow
$ # docker-compose.dev.yml is a small file which overrides some service parameters for docker-compose.yml. 
$ # This way the core of the config is shared, yet we can flip some services to use the dev images, mount source into the containers etc.
$ docker-compose -f docker-compose.yml -f docker-compose.dev.yml up
$ # everything will be built, started, migrated and started in the dev local mode ready for immediate dev usage.

However you should really just use the provided and newly upgraded ./dev.sh script, which by default will start the dev env, attach into the running containers and make sure the containers are running as your actual user so the filesystem wont be messed up potentially (compared to the containers running as root and then changing your mounted folders!).

It makes the attaching experience super nice as it will:

• Run docker logs in the new tab opened so you can see everything that has happened in the container so far
• The tabs are configured so that you can press ctrl-c to stop the process running inside the container leaving you in a bash session inside the container, but not stop the container itself. This is useful in dev as you often want to stop the dev server, run a management command and quickly restart it etc.
• Finally the bash you are left in after pressing ctrl-c will have a history populated, so you can press up to get the command that the container was running before you pressed ctrl-c!

$ ./dev.sh # same as the up command above but also ensures the containers run as the running user!
$ ./dev.sh --build # ups and rebuilds
$ ./dev.sh restart # stops and then ups
$ ./dev.sh restart --build # stops, builds, ups
$ ./dev.sh build_only # just builds
$ ./dev.sh dont_attach # does not create tabs and attach to the containers at the end
$ ./dev.sh dont_attach restart --build # You can combine multiple arguments like so
$ ./dev.sh dont_migrate # ups but doesn't migrate automatically on startup
$ ./dev.sh dont_migrate dont_sync dont_attach restart --build # even more flags!
# Any commands found after the last `./dev.sh` command will be passed to the `docker-compose up` call made by dev.sh
# This lets you say do --build on the end or any other docker-compose commands using dev.sh!
$ ./dev.sh restart {EXTRA_COMMANDS_PASSED_TO_UP}  
$ ./dev.sh stop # stops
$ ./dev.sh kill # kills (the old stop_dev.sh) 
# Bind to different ports on the host manage incase you are already running them and they clash! (also works with just docker-compose up)
$ POSTGRES_PORT=5555 REDIS_PORT=6666 MJML_PORT=7777 ./dev.sh 

Using the Docker Images directly

There are now 4 docker images in baserow:

• baserow_backend (Found in backend/Dockerfile)
  • This by default starts up with gunicorn running and can be configured via environment variables to migrate / sync_templates on startup
  • This image also is used for the celery container. By passing the command "celery" to this image it instead starts up celery.
  • This image only installs production dependencies.
• baserow_backend-dev (Found in backend/docker/Dockerfile.dev)
  • This by default starts up with django's runserver, also configurable like the above
  • This image also installs the dev dependencies
  • A separate celery-dev command is supported to startup hot-restarting celery for dev mode.
  • Otherwise it is identical to the baserow_backend image
• baserow_web-frontend (Found in web-frontend/Dockerfile)
  • This builds nuxt for production as part of the image itself and by default starts a prod nuxt server on startup
  • It only installs production dependencies
• baserow_web-frontend-dev (Found in web-frontend/docker/Dockerfile.dev)
  • This does not do a production build in the image
  • By default it starts a nuxt dev server
  • It contains dev dependencies

The web-frontend and backend docker images now come with a set of commands you can use:

Usage: docker run baserow_backend COMMAND
Commands
local     : Start django using a prod ready gunicorn server
dev       : Start a normal Django development server
bash      : Start a bash shell
manage    : Start manage.py
python    : Run a python command
shell     : Start a Django Python shell
celery    : Run celery
celery-dev: Run a hot-reloading dev version of celery
lint:     : Run the linting
help      : Show this message

Usage: docker run baserow_web-frontend COMMAND
Commands
dev      : Start a normal nuxt development server
local    : Start a non-dev prod ready nuxt server
lint     : Run the linting
lint-fix : Run eslint fix
bash     : Start a bash shell
help     : Show this message

The default commands used on a per image basis are :

• Web Frontend
  • Dev Image = 'dev'
  • Local Image = 'local'
• Backend
  • Dev Image = 'dev'
  • Local Image = 'local'

Other Features / Fixes included in this MR

• The docker images build much faster because:
  • We no longer install postgres in the backend images
  • The image layers have been re-ordered such that changing the code alone will still used the cached layers containing the installed requirements, whereas before changing the code would always trigger the docker build to do a full requirements install.
• The docker images are smaller
  • We have switched to the slimmer variant of the python image
  • We perform clean-up of unneeded packages in the images
• The docker containers no longer run as root
  • In dev mode using ./dev.sh or by manually specifying the env variables CURRENT_UID and CURRENT_GID you can run the containers as your own user account, stopping the images accidentally screwing up your mounted source folders as root.
  • In local mode we just create a non-dev user and use that instead of root by default as recommended for security reasons.
• When appropriate we use a special init process which runs as pid 1 in our images.
  • This prevents zombie processes
  • This init process correctly handles signals
• In the backend image when needed we wait for postgres to be available before starting up the django server / running migrations
• The docker images used on their own by default will not migrate or sync on startup. This is opt in by setting the MIGRATE_ON_STARTUP or SYNC_TEMPLATES_ON_STARTUP environment variables to "true". This is done by default in the dev.sh file or when you just using the docker-compose files.
• The unused mysqlclient python requirement which caused install problems has been removed
• When EMAIL_SMTP is not set we default to the django console.EmailBackend (perhaps this should massively warn in non-dev mode!)
• We now use a fixed verision of liminspace/mjml-tcpserver instead of the latest to prevent unintended image upgrades
• You no longer ever need to manually create a network
• When using the docker-compose files postgres will have it's state persisted on a named docker volume
• You can now run the baserow env easily alongside existing services by changing the external ports via env variables (say you have a postgres db already running locally just do POSTGRES_PORT=5543 docker-compose up so they don't clash!)