Edgy-Camille version
Big improvements again on the EdgeSets, HornNets, MarionNets, megarun3, and pipeline side of things. Notably:
- HornNet
* Quickslicer and training improvements
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/24
- quickslicer now also outputs abs abs thresh mismatch and the mean of all measures
- target data for multi-output NNs can be scaled with mean of stds
- hyperparameters can be tracked in TensorBoard for hyperparameter scans
- load_from_store without columns (columns=False)
- repeating feature_names in megaHornNet
- Test and build pipeline
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/26
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/33
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/35
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/36
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/37
* Add full CI test pipeline for Python3.6 and Python3.8. Target Python is
* Python 3.7. (999737d4, dc0a4876, 534366c5, da6bad29, 5eb17813)
* Use conda-friendly packaging using .txt files (b13f7218)
* Use setuptools_scm to set the version of the package (e6ab479c)
* Do not upload full venv as job artifact (586d99e8) [thanks Francis noticing file system explosion]
* Require at least TensorFlow 2.2.0 to run Dan Nieuwenhuizens workflows on m100 (b13f7218)
* Replaced ye olde mega_nn.py and replaces it with a new and shiny qlknn_hyper_10D.py. See a7070727
- More flexible clipping (e104fc52, c73798d4)
- Allow momentum filter to allow for negative values (a6adefee)
* Drops support for Python 2.7 (f8bb4cff)
* Blackify.py files:
- setup.py (e0537d6b)
- All python files (c4277fd8, a6adefee)
- All python scripts (35a9019c, 90338421)
* Blackify ignore list can be found in .git-blame-ignore-revs, see (01bb3a72, 1c990405)
* Upgrades Black to use pyproject.toml and use python 3.7 syntax
* Creates a new global fixture store_path build from
* qlk_h5_gen5_4D_dataset for testing of gen5-aware code
* Moves the existing tests to use above fixture
* Adds a silly echo function to qlknn/training/keras_models.py:QLKNet to
* test partial initialization of Keras classes
* Adds initial mostly empty tests for Camille's edgerun code
* Adds fallbacks to outlier dropping
* Build packages on every tag
- Implement Sphinx documentation and CI for qlknn package
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/27
* Autoparse requirements*.txt (1d26f946)
* Clean up package dependencies (1d26f946, 1ecf2b71)
* Use scripting over YAML constructs (1ecf2b71, 1ecf2b71)
* Use Sphinx instead of Doxygen (126880b9)
- Add workaround for pip install bug
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/29
* See https://github.com/pypa/pip/issues/7953 relating to editable installs
- Quickslicer cleanup
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/30
* Move CLI interface from docopt to argparse (88615aee, dc5f6dd1)
* Make qlknn_hyper_10D.py usable in the quickslicer (c3445450)
* Allow slicing with the "wrong" dataset {e.g. 9D and 4D for 7D networks} (7decf15b, 127ac2e2)
* Add quickslicer as console script (5a30f06a)
* Reduce reliance on global variables (9e2403be)
* Move testdata to dedicated folder (4ef94768)
* Disable luigi tests (7d378723)
* Add tests and testdata for
- quickslicer (3d459e7b, 090aa7dd, 7ae1e47a)
- committees (d8bee41a)
- FFNNs (08d7031e, 9aa23dee)
* Shuffle the dataset is now the default (1d31cfb2)
- Clustering scripts by @BartKremers and @aaronkho
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/28
- MarionNets
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/31
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/32
preparing for divnet training for QLKNN-11D and MarionNets:
* pytest test suite improvements
- General cleanup (c0cd2481)
- Add tests for divsum variables in dataset (fcd5f92e, bdd19f19, 7622d5bb)
- Removal of unused tests (7c5b6ad9, 6df4b1b7)
- Add new "CLI-like python code" tester (9e8db883, 928c511c, 0ed9c395)
* qlknn refactor
- Move all code to Black codestyle (4f43f45c, 60451e24), see https://black.readthedocs.io
- Tiny internal syntax improvements (c822debe)
- WIP code to misc (4eeb8283)
* Add more entry points for CLI tools (7002755b, a74bbeb2)
* Allow generating divsum variables on the fly for QLKNN training (9359d21b, ce1c14b2, 1792a38b, d3b63ed1, c9b35831)
* Improvements to docs (d7d30297)
* Karel-style linter settings (d60d4fd1)
* Improved indexing and low/high bounds for get_output (6d1de645), thanks @MESmedberg
- CamilleNets
https://gitlab.com/qualikiz-group/QLKNN-develop/-/merge_requests/34
* Implements all contributed CamilleNet scripts. Thanks @cambouvy