BETSE 1.1.0 (Nice Nestor) released.
Significant changes include:
* "--headless" option. The "betse" command now accepts a "--headless"
option, permitting both end users and our test suite alike to
explicitly enable headless operation. While the codebase did already
detect most headless environments and thus automatically enable
headless operation as needed, permitting interested parties to do so
sanitizes numerous use cases.
* Backward compatibility guarantee extended to BETSE >= 0.5.0. BETSE now
guarantees backward compatibility with simulation configurations
supported by relatively recent versions of BETSE (i.e., BETSE 0.5.0
and newer).
* PyYAML -> ruamel.yaml >= 0.15.24 required. BETSE now requires the
comment- and whitespace-preserving "ruamel.yaml" YAML roundtripper
rather than the antiquated PyYAML parser. Inadequacies in the Python
installation ecosystem (e.g., setuptools, wheels) prevent BETSE from
permissively accepting one of several different YAML implementations
at installation time. Since "ruamel.yaml" is the superset of all YAML
implementations, BETSE 1.1.0 officially drops PyYAML in favour of
"ruamel.yaml".
* YAML 1.1 -> 1.2. All bundled YAML files have been trivially bumped
from compliance with YAML 1.1 to 1.2 (i.e., the most recent YAML
standard), resolving numerous valid complaints from "ruamel.yaml" at
load time. To preserve backward compatibility with prior
YAML-formatted simulation configuration files, files compliant with
YAML 1.1 are implicitly interpreted as compliant with YAML 1.2
instead.
* Mandatory dependencies bumped:
* numpy >= 1.13.0. A recent version of numpy is now required, due to
our widespread usage of the optional "axis" keyword argument passed
to the numpy.unique() function first introduced by numpy 1.13.0.
Doing so resolves a recent spate of CI failures in Windows-ish
AppVeyor pipelines.
* py.test >= 3.7.0. A recent version of py.test is now required, which
introduced a new "package" scope for fixtures. This scope is
required to efficiently initialize and deinitialize application
metadata singletons for unit tests.
* Deprecation warnings resolved:
* A deprecation from the "betse" script wrapper.
* A deprecation from the numpy.unique() function on failing to
explicitly pass the "axis" keyword argument.
* Numpy optimization detection. To improve detection of optimized Numpy
installations (i.e., of a "numpy" package linked against a
multithreaded BLAS implementation), application startup preferentially
detects whether the active Python interpreter is managed by any of the
following, in which case Numpy is guaranteed to be optimized:
* The "conda" package manager.
* The modern Gentoo Linux scientific stack (i.e., "eselect-ldso").
* Problematic matplotlib backends blacklisted (i.e., prohibited):
* Unconditionally blacklisted all Qt 4-specific matplotlib backends
(e.g., "Qt4Agg", "Qt4cairo"). The Qt Company ceased maintenance of
Qt 4 in 2015, implying Qt 4 to be insecure, fragile, and largely
unavailable.
* Conditionally blacklisted all Qt 5-specific matplotlib backends
(e.g., "Qt5Agg", "Qt5cairo") under headless environments. When
headless, these backends reliably cause Qt 5 to silently fail with a
segmentation fault.
* Logging improvements:
* Logfile rotation race conditions resolved. A long-standing issue
inviting race conditions between multiple processes contending for
write access to rotational logfiles has now been resolved, thanks to
intrepid end user Edward Ned Harvey at the Levin Lab. Thanks a
metric ton, Ed!
* Logfile closure. Open logfile handles are now guaranteeably closed
in a platform-portable manner at application shutdown, resolving
Windows-specific "ResourceWarning: unclosed file" warnings. Namely:
* The application metadata singleton is now unconditionally
deinitialized at application shutdown (even in the event of fatal
exceptions).
* Our default logging configuration now removes all existing
handlers from the root logger and close all open file handles
maintained by those handlers at both initialization and
deinitialization time.
* Testing improvements:
* "${DISPLAY}" squelched. The X.org-specific "${DISPLAY}" environment
variable is now squelched during testing, resolving functional test
failures.
* Windows-specific warnings logged as informational non-warnings, a
crude workaround to prevent PowerShell from erroneously raising
non-human-readable exceptions on the first write to standard error.
(Why!?)
* "--export-sim-conf-dir" removed. The awkward "--export-sim-conf-dir"
option previously accepted by our test suite has been obsoleted.
* Our AppVeyor configuration now:
* Locally resolves conda/conda#8836, a recent Windows-specific
Anaconda change significantly breaking backward compatibility with
most existing AppVeyor configurations -- including ours. To do so,
the base Anaconda environment is now explicitly activated in an
early-time manner.
* Activates its "conda" environment via the "activate" command
rather than the "conda activate" subcommand, which appears to be
non-functional.
* Coerces high-level PowerShell stream objects produced by running
the "py.test" command into low-level strings. If omitted,
PowerShell insanely interprets the first write to stderr as a
non-human-readable exception.
* Our GitLab-CI Runner configuration now:
* Caches both "conda" environments and packages to project-local
directories, which should reduce redundant pipeline overhead.
* Imports environment variables required for sane "conda" usage and,
in particular, the recently introduced "conda activate"
subcommand.
* Has been streamlined for porting to downstream consumers (e.g.,
BETSEE):
* Removed all disabled configuration pertaining to "pip".
Leveraging "pip" from a Miniconda-based configuration is
fundamentally wrong.
* Printed application metadata before running the test suite via
the new "--headless" option.
* Documentation improvements:
* Docstring revisions for numerous subclasses in the
"betse.science.config.export.visual" submodule for accuracy.
* API generalizations:
* Renamed the prior:
* "betse.metaapp" submodule to "betse.appmeta".
* "betse.util.app.meta.metaappabc" submodule to
"betse.util.app.meta.appmetaabc".
* "betse.util.app.meta.appmetadep" submodule to
"betse.util.app.meta.appmetamod".
* "betse.util.app.meta.metaappton" submodule to
"betse.util.app.meta.appmetaone".
* "betse.util.io.log.logconfig" submodule to
"betse.util.io.log.conf.logconf".
* "betse.util.io.log.logformat" submodule to
"betse.util.io.log.conf.logconfformat".
* "betse.util.io.log.loghandle" submodule to
"betse.util.io.log.conf.logconfhandle".
* "betse_test.fixture.metaapper" submodule to
"betse_test.fixture.initter".
* Generalized the existing "betse.exceptions" submodule:
* Defines new exception classes, including:
* "BetseMappingKeyException".
* "BetseLogRaceException", unique to the aforementioned log
rotation race condition.
* Generalized the existing "betse.metadata" submodule:
* Replaced all references to the obsolete static "SCRIPT_BASENAME"
global with the dynamic
betse.util.app.meta.appmetaone.get_app_meta().package_name()
property, which properly applies to downstream consumers (e.g.,
BETSEE).
* Renamed the "GIT_TAG_OLDEST_BACKWARD_COMPATIBILITY" global to
"GIT_TAG_COMPAT_OLDEST" for brevity.
* Reverted this global to "v0.5.0", restoring guaranteeable backward
compatibility with all simulation configurations from BETSE >=
0.5.0.
* Generalized the existing "betse.lib.matplotlib.matplotlibs"
submodule:
* Defined a new MplConfig.logger() property, yielding Matplotlib's
root logger object (i.e., "matplotlib").
* Optimized the existing "betse.lib.numpy.numpys" submodule:
* Defined a new _is_blas_optimized_conda() tester, deferring to the
newly defined betse.util.py.pys.is_conda() tester.
* Optimized the existing is_blas_optimized() tester to
preferentially call the _is_blas_optimized_conda() tester first,
which is guaranteed to be both the most optimal and portable
solution.
* Streamlined the existing "betse.lib.setuptools.command.supcmdtest"
submodule:
* Removed the obsolete "test.export_sim_conf_dir" instance variable.
* Generalized the existing "betse.lib.yaml.yamls" submodule:
* Refactored the load() function to accept a new optional
"yaml_version" parameter, overriding any version directive
prefacing the passed YAML-formatted file. When passed such a
parameter with a value other than "1.1" (e.g., "1.2"), this
function temporarily ignores all non-fatal warnings emitted by
"ruamel.yaml" specific to the YAML 1.1 specification.
* Defined a new "betse.lib.yaml.abc.yamlfileabc" submodule:
* Shifted the prior "betse.lib.yaml.abc.yamlabc.YamlFileABC"
superclass into this submodule:
* Defined a new copy() method, encapsulating the lower-level
load() and save() methods for safe copying YAML-formatted files.
* Generalized the load() method to accept variadic keyword
arguments, passed as is to the betse.lib.yaml.yamls.load()
function.
* Generalized the save() method to accept two new optional
parameters:
* "is_conf_file_overwritable", the overwrite policy for the
passed target file.
* "conf_subdir_overwrite_policy", the overwrite policy for
subdirectories of this file.
* Defined a new "YamlFileDefaultABC" superclass defining:
* An abstract conf_default_filename() class property, returning
the absolute filename of the subclass-specific default YAML file
(e.g., default simulation configuration).
* A concrete copy_default() method, copying this file to the
passed target filename.
* Removed the "betse.science.config.confio" submodule.
* Generalized the existing "betse.science.parameters" submodule:
* Refactored the "Parameters" class:
* To subclass the newly defined "YamlFileDefaultABC" subclass.
* Refactored the load() method to override any version directive
prefacing all YAML-formatted files loaded by this method with
"1.2", preserving backward compatibility with prior simulation
configuration files erroneously prefaced by the "%YAML 1.1"
directive.
* Replaced all prior calls to the
betse.science.config.confio.write_default() function with calls to
the save_default() method.
* Generalized the existing "betse.science.math.mesh" submodule:
* Improved the DECMesh.refine_mesh() method to log rather than
merely print informational messages.
* Generalized the existing "betse.util.app.meta.metaappabc" submodule:
* Refactored the "AppMetaABC" superclass:
* Defined a new deinit() method, deinitializing both this
application metadata singleton and the current application.
* Defined the following new properties:
* module_metadata(), the subclass-specific application metadata
submodule.
* module_metadeps(), the subclass-specific application
dependency metadata submodule.
* test_package(), the root test package for the current
application if found or raises an exception otherwise.
* test_dirname(), the absolute dirname of that package's
directory if found or raises an exception otherwise.
* test_data_dirname(), the absolute dirname of that package's
data subdirectory if found or raises an exception otherwise.
* Replaced all prior references to the prior abstract
betse.util.cli.cliabc._module_metadata() property to the new
module_metadata() property and removed the former.
* Generalized the existing "betse.util.app.meta.appmetamod" submodule:
* Defined a new merge_module_metadeps() function:
* Dynamically creating and returning a new application dependency
metadata module with arbitrary name iteratively merged from the
contents of all passed application dependency metadata modules.
* Copying the "RequirementCommand" class globally defined by the
first such module into the output module.
* Generalized the existing "betse.util.app.meta.appmetaone" submodule:
* Defined a new deinit() function, conditionally encapsulating the
lower-level betse.util.app.meta.appmetaabc.AppMetaABC.deinit()
method.
* Generalized the "betse.util.cli.cliabc" submodule:
* Refactored the _make_options_top() method into an "_options_top"
property, preserving orthogonality with the existing
"betse.util.cli.clicmdabc._subcommander_top" property.
* Refactored the "_is_option_matplotlib_backend" boolean property
into an "_matplotlib_backend_name_forced" string property,
enabling subclasses to force initialization of a desired
subclass-specific matplotlib backend.
* Defined a new "betse.util.io.log.conf" subpackage.
* Defined a new "betse.util.io.log.conf.logconfcls" submodule:
* Shifted the prior "betse.util.io.log.logconfig.LogConfig" class to
"LogConf" in this submodule.
* Generalized the existing "betse.util.io.log.conf.logconf" submodule:
* Defined a new deinit() function calling a new
betse.util.io.log.conf.logconfcls.LogConf.deinit() method,
deinitializing the initialized logging configuration and hence
closing all open file handles associated with this configuration.
* Improved the existing init() function to reduce to a noop rather
than reinitialize a previously initialized logging configuration.
* Generalized the existing "betse.util.io.log.conf.logconfhandle"
submodule:
* Refactored the .LogHandlerFileRotateSafe" subclass:
* Revised class and method docstrings for clarity, including a
recommendation that external callers manually pass the
"--log-file" CLI option when running concurrent BETSE processes.
* Refactored the _emit_safely() method to:
* Accept a "self" parameter, a critical bug fix.
* Sleep for 100ms rather than 50ms.
* Print additional debug output to standard error documenting
this method's automatic resolution process.
* Raise a human-readable exception on exhausting all other
options, including the same recommendation as documented in
docstrings.
* Generalized the existing "betse.util.io.log.logs" submodule:
* Renamed the get() function to get_logger() for disambiguity.
* Generalized the existing "betse.util.io.stdouts" submodule:
* Repaired the output_lines() function, which suffered a critical
(albeit trivial) defect.
* Defined a new "betse.util.os.brand.linux" submodule:
* Defined a new is_mir() tester, returning true only if the active
Python interpreter is running under the Mir compositor.
* Defined a new is_wayland() tester, returning true only if the
active Python interpreter is running under the Wayland compositor.
* Shifted the existing betse.util.os.is_linux() tester into this
submodule.
* Defined a new "betse.util.os.brand.macos" submodule:
* Shifted the existing betse.util.os.is_macos() tester into this
submodule.
* Defined a new "betse.util.os.brand.posix" submodule:
* Defined a new is_x11() tester, returning true only if the active
Python interpreter is running under an X11 display server.
* Shifted the existing betse.util.os.is_posix() tester into this
submodule.
* Generalized the existing "betse.util.os.brand.windows" submodule:
* Defined a new is_version_10_or_newer() tester, returning true if
the current platform is Windows >= 10.
* Defined a new get_api_version() getter, returning the current
low-level Windows API (WinAPI, Win32) version.
* Shifted the is_windows(), is_windows_cygwin(),
is_windows_vanilla(), and is_windows_wsl() testers from the
existing "betse.util.os.brand.os" submodule into this submodule.
* Generalized the existing "betse.util.os.displays" submodule:
* Defined a new is_dpi_scaling() tester, returning true if the
current platform natively supports high-DPI scaling.
* Refactored the is_headless() tester to defer to the most recent
call to the newly defined set_headless() function (if any).
* Defined a new set_headless() setter, enabling callers to
explicitly force headless operation and hence override the
implicit detection of headless environments performed by the
is_headless() tester.
* Generalized the existing "betse.util.os.kernels" submodule:
* Defined a new is_version_greater_than_or_equal_to() tester,
enabling callers to trivially detect minimum kernel versions and
hence the existence of platform features introduced by those
versions.
* Revises the get_version() getter to return the high-level Windows
kernel version (e.g., "10.0.10240") rather than the low-level
Windows API version (e.g., "6.2.9200").
* Generalized the existing "betse.util.os.shell.shellenv" submodule:
* Defined a new to_str() function, returning a human-readable string
of all environment variables defined by the host shell
environment.
* Generalized the existing "betse.util.py.module.pymodname" submodule:
* Defined a new make_module() function, enabling callers to
dynamically generate new in-memory modules containing arbitrary
module attributes.
* Generalized the existing "betse.util.py.module.pymodule" submodule:
* Defining a new is_module() tester, returning true only if the
passed object is a module.
* Generalized the existing "betse.util.type.obj.objects" submodule:
* Refactoring the get_attr() getter to embed the fully-qualified
name of the passed module when passed a module in raised
exceptions.
* Generalized the existing "betse.util.py.pys" submodule:
* Defined a new is_conda() tester, returning true if the active
Python interpreter is managed by conda. For debuggability, this
metadata is now logged when the "info" subcommand is run.
* Defined a new "betse.util.test.pytest.fixture" subpackage:
* Defined a new "betse.util.test.pytest.fixture.pytfixture"
submodule:
* Defined a new monkeypatch_session() fixture, generalizing the
standard function-scope "monkeypatch" fixture to session scope.
* Generalized the existing "betse.util.test.pytest.pytests" submodule:
* Defined a new output() function, printing the passed objects to
standard output in a format consistent with that of standard
"py.test" messages.
* Generalized the existing "betse.util.type.iterable.iterables"
submodule:
* Defined a new to_str() function, synthesizing a human-readable
prettified string of the passed iterable.
* Improves the existing to_iterable() converter to behave as
expected when passed an input generator.
* Defined a new "betse.util.type.iterable.iteriter" submodule:
* Defined a new iter_pairs() iterator, creating and returning a new
iterable of all possible pairs of items in the passed iterable.
* Defined a new iter_combinations() iterator, creating and returning
a new iterable of all possible k-combinations of items in the
passed iterable.
* Generalized the existing "betse.util.type.iterable.mapping.mappings"
submodule:
* Defined a new die_unless_keys_unique() function, raising an
exception unless no passed dictionaries collide (i.e., contain the
same key).
* Defined a new is_keys_unique() tester, returning true only if no
passed dictionaries collide.
* Defined a new iter_keys_ascending() iterator, returning an
iterable of all keys of the passed dictionary in ascending order
(typically, lexicographic).
* Removed the unused (and largely useless) format_map() function,
now superceded by the newly defined
betse.util.type.iterable.iterables.to_str() function.
* Defined a new "betse.util.type.iterable.mapping.mapmerge" submodule:
* Shifted the betse.util.type.iterable.mapping.mappings.merge_maps()
function into this submodule.
* Defined a new "MergeCollisionPolicy" enumeration type whose
members each describe a different type of key collision merger
policy (i.e., strategy for merging keys shared by one or more
mappings).
* Refactored the merge_maps() function to:
* Merge arbitrarily many dictionaries under any key collision
policy.
* Behave as expected when passed a non-sequence of input mappings.
* Defer to the newly defined
betse.util.type.iterable.mapping.maptest.die_if_maps_collide()
validator.
* Defined a new "betse.util.type.iterable.mapping.maptest" submodule:
* Shifted all testing and exception handling logic from the
"betse.util.type.iterable.mapping.mappings" submodule into this
submodule.
* Renamed the following functions for orthogonality:
* die_unless_keys_equal() to die_unless_maps_keys_equal().
* die_unless_keys_unique() to die_unless_maps_keys_unique().
* is_key() to has_keys().
* is_keys_equal() to is_maps_keys_equal().
* is_keys_unique() to is_maps_keys_unique().
* Defined a new die_unless_has_keys() validator, raising an
exception unless the passed mappings contains all passed keys.
* Replaced the largely useless die_unless_maps_keys_unique()
validator with a new die_if_maps_collide() validator, raising an
exception only if no passed mappings collide.
* Replaced the largely useless is_maps_keys_unique() tester with a
new is_maps_collide() tester, returning true only if one or more
passed mappings collide (i.e., contain key-value pairs containing
the same keys but differing values).
* Refactored the die_unless_maps_keys_unique() validator to improve
the granularity of the exception message raised by this function,
which now iteratively finds the exact set of key collisions in the
passed dictionaries with the newly defined
betse.util.type.iterable.iteriter.iter_pairs() iterator.
* Refactored the has_keys() tester to accept a simple iterable of
all keys to be tested rather than a variadic positional tuple of
such keys.
* Generalized the "betse.util.type.iterable.sequences" submodule:
* Defined a new die_if_length_less_than() validator, raising an
exception if the passed sequence contains less than the passed
number of items.
* Defined a new to_sequence() converter, converting the passed
iterable into a sequence as needed.
* Refactored the die_if_empty() validator to operate similarly to
all other validators defined by this submodule.
* Generalizes the "betse.util.type.iterable.mapping.maptest"
submodule:
* Refactored the is_maps_collide() tester to call the newly defined
betse.util.type.iterable.sequences.die_if_length_less_than() and
betse.util.type.iterable.set.sets.make_union() functions, thereby
both simplifying and correcting the existing implementation.
* Defined a new "betse.util.type.iterable.set" subpackage:
* Shifted the existing "betse.util.type.iterable.sets" submodule to
"betse.util.type.iterable.set.setcls" for disambiguity.
* Defined a new "betse.util.type.iterable.set.sets" submodule:
* Defined a new make_union() function, calculating the union of
all items in arbitrarily many possibly non-set iterables.
* Generalized the existing "betse.util.type.text.string.strs"
submodule:
* Refactored the get_prefix_preceding_char_or_none() getter:
* Renamed this getter to get_prefix_or_none().
* Renamed the "char" parameter accepted by this getter to
"anchor".
* Added an optional "is_first" parameter to this getter, enabling
callers to retrieve both greedy and non-greedy prefixes.
* Streamlined the existing "betse_setup.buputil" submodule by removing
extraneous functionality (e.g., die_unless_module(), is_module()).
* Streamlined the existing "betse_test.conftest" submodule:
* Removed all functionality pertaining to the ${DISPLAY} environment
variable from the less robust and reusable pytest_configure()
function in favour of the
betse_test.fixture.initter.betse_autouse() fixture.
* Removed the obsolete "EXPORT_SIM_CONF_DIRNAME" global.
* Defined a new "betse_test.data" subdirectory embedding the default
simulation configuration supplied by BETSE 0.5.0, enabling the
betse_test.func.sim.test_sim.test_cli_sim_compat() functional test
to fully exercise backward compatibility with this configuration.
* Rewrote the "betse_test.fixture.initter" submodule:
* Defined a new autouse package-scoped betse_init_package() fixture,
intended to be imported by the "betse_test.unit.conftest" plugin
to guarantee application initialization before unit testing.
* Renamed the betse_meta_app() fixture to betse_autouse(), which now
additionally unsets the ${DISPLAY} environment variable if set.
* Moved the existing betse_test.conftest._init_app() and
_deinit_app() functions into this submodule as init_app() and
deinit_app() for importability elsewhere.
* Generalized the existing "betse_test.fixture.simconf.simconfer"
submodule:
* Redefined the betse_sim_conf() fixture in terms of the lower-level
betse_sim_conf_default() fixture.
* Refactored the betse_sim_conf_compat() fixture to leverage the
Git-hosted "betse_test/data/v0.5.0/yaml/sim_config.yaml"
simulation configuration corresponding to the default
configuration circa BETSE 0.5.0.
* Generalized the existing "betse_test.fixture.simconf.simconfclser"
submodule:
* Refactored the "SimConfTestABC" superclass to initialize the
application metadata singleton before saving a simulation
configuration for the current functional test.
* Refactored the "SimConfTestInternal" subclass to require
general-purpose "src_conf_filename" and "trg_conf_filepath"
filenames be passed on initialization. Previously, this subclass
only accepted the latter and defaulted the former to the default
simulation configuration file.
* Generalized the existing "betse_test.fixture.simconf.simconfwrapper"
submodule:
* Refactored the "SimConfigTestWrapper" class:
* Refactored the __init__() method to accept a high-level
"Parameters" object rather than a low-level simulation
configuration filename.
* Removed a number of extraneous methods and properties (notably,
the make_default() class method).
* Generalized the existing "betse_test.func.fixture.clier" submodule:
* Called the betse.util.app.meta.metaappabc.AppMetaABC.deinit()
method before running each functional test.
* Removed the obsolete "betse_test.func.sim.test_sim_export"
submodule.
* Generalized the existing "betse_test.func.sim.test_sim" submodule:
* Removed all @skip decoration from the test_cli_sim_compat() test.
* Defined a new "betse_test.unit.type.iterable.mapping.test_mapmerge"
submodule, fully exercising the newly generalized
betse.util.type.iterable.mapping.mapmerge.merge_maps() function (now
featuring 667% more obscure Cthulhu mythos references).
* Defined a new "betse_test.unit.app.meta.test_appmetamod" submodule,
fully exercising the newly defined
betse.util.app.meta.appmetamod.merge_module_metadeps() function.
* Generalized the betse_test.unit.path.test_dirs.test_packages_init()
unit test to exclude all data-specific directories from
consideration, including both the "AppMetaABC.data_dirname" and
"AppMetaABC.test_data_dirname" directories if found.
* Generalized the existing
betse_test.unit.type.iterable.test_iterables.test_to_iterable() unit
test to exercise additional edge cases of that underlying function.