Skip to content

Investigate other ETL approaches

We should look at other ETL frameworks (both PHP and otherwise) and see what good ideas might be applicable to soong.

As a starting point, look at ETL packages on packagist (PHP) and the Awesome ETL list (non-PHP).

Let's treat this issue as a meta and open a separate issue for each library/framework/platform investigated.

Edit: On second thought let's make a list here of PHP ETL/migration packages on Packagist, and non-PHP frameworks. In most cases, a comment here and striking out the package/framework should suffice - only create a separate issue where a given package/framework has some particularly interesting ideas.

PHP packages

Stars/DLs Package Repo Status
163/134859 portphp/portphp https://github.com/portphp/portphp See soongetl/architecture#15 (moved)
83/3819 knplabs/etl https://github.com/docteurklein/php-etl Nothing we can use
71/632 tdt/core https://github.com/tdt/core Not relevant
60/7398 marquine/php-etl https://github.com/leomarquine/php-etl Nothing we can use
29/261 umpirsky/extraload https://github.com/umpirsky/Extraload Nothing we can use
19/2605 bentools/etl https://github.com/bpolaszek/bentools-etl
12/1161 bisight/etl https://github.com/bisight/etl
8/4781 cleverage/process-bundle https://github.com/cleverage/process-bundle
6/9136 fab2s/yaetl https://github.com/fab2s/YaEtl
4/1381 tdt/input https://github.com/tdt/input
3/3543 bookeen/etl-workflow https://github.com/BOOKEEN/etl-workflow
2/57290 giift/etl https://github.com/giift/etl
1/20 iu-redcap/redcap-etl https://github.com/IUREDCap/redcap-etl
0/282 milesasylum/slurp https://github.com/courtney-miles/slurp
0/166 yevgen-grytsay/etl-suite https://github.com/yevgen-grytsay/etl-suite
0/121 xamplifier/etl https://github.com/xamplifier/etl
0/117 cyve/etl https://github.com/cyve/etl
0/98 inl/etl-processor https://github.com/mpierzchalski/etl-processor
0/95 smartbooster/etl-bundle https://github.com/smartbooster/etl-bundle
0/44 kiboko/etl https://github.com/kiboko-labs/etl
0/33 ralphschindler/etl-pipeliner https://github.com/ralphschindler/php-etl-pipeliner

Note the above come from searching packagist.org for "etl" (plus PortPHP suggested by @weitzman in https://gitlab.com/soongetl/soong/issues/8 - I've also searched on "migration", but at least among the first couple of pages of results they look pretty much to be about database migrations (e.g., changing schemas between different versions of a framework) and not applicable to this project.

Edited by Mike Ryan
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information