Skip to content

Improved developer experience with docker-compose ft. massive refactor

Austin Pray requested to merge austinpray/meltano:docker-next into master

Take two of !52 (merged)

I made a screencast that goes over all these changes and the thought process behind them:

UPDATES

See these two comments about running Meltano outside of docker

High level

  • Booting a demo version of the app is extremely simple. Down to three commands: https://gitlab.com/austinpray/meltano/blob/efd418e13f7504467599d80740b1d003afd601bd/README.md#how-to-install-and-run-meltano
  • The build process uses Make so the build command api is super simple. Just run make and only the parts of the app that have changed will be rebuilt.
  • Improvements to the production docker setup. This dev docker-compose setup uses images that are deployable to production. Dev like you prod.
  • Meltano is now pip-installable.
  • Meltano uses the python packaging system to resolve plugins and stuff instead of a bespoke system
  • Removed a ton of assorted jank such as
    • python coding errors
    • misunderstandings of python's import and module system
    • merge conflicts that were committed to master
    • files that were gitignored but checked into the repo

If you want to know more about what commands are available to orchestrate the dev environment you can just look at the Makefile or run make explain_makefile and then the following will be available in your browser: image

Next Steps

We need to review again and iron out how this plays into the overall meltano strategy.

Edited by Melty Bot

Merge request reports