[Feature Request] Interactive REPL + Non-interactive Scripting
This feature request comes courtesy @dglmoore, who implemented information theoretic-specific support in his BETSE fork for both:
- An interactive Python read–eval–print loop (REPL) integrated with BETSE.
- Non-interactive Python scripting integrated with BETSE.
This issue is intended to serve as the fertile charnel grounds for an eventual general-purpose solution to both features – which, of course, are intimately related. Discussion points along the road to feature-complete peace include:
Integration – What's That Mean?
What exactly does the ambiguous phrase "integrated with BETSE" mean? I don't know and, honestly, neither do you. To begin imagining all the beautiful integrations that could be, consider:
- Integration that pre-imports critical first-party (i.e., BETSE) and third-party (i.e., non-BETSE) modules of interest. Candidates include:
-
import betse as be
, just 'cause. -
from betse.science.parameters import Parameters
, simplifying configuration. -
from betse.science.sim import Simulator
, simplifying simulation. -
from betse.util.io.log import logs
, simplifying logging. -
import matplotlib.pyplot as plt
, simplifying plotting. -
import numpy as np
, simplifying... Numpying. _It's a word!_
-
Unlike Numpy, there doesn't appear to be a single SciPy package or module of interest. Hence, this list intentionally omits scipy
. Likewise, the following BETSE subpackages and submodules are under significant renovation and (…arguably) unsafe for external access by users or scripts:
-
betse.science.plot
. -
betse.science.filehandling
.
With great reluctance and interminable sighing, both are omitted. (For now.)
Anything else? Hit me with your best import list.
Is This Really a Good Idea?
I am Clueless Joe. Implicitly importing packages and modules on behalf of users and scripts introduces a number of not-so-subtle issues that typically annoy me when other libraries who shall remain nameless (PyInstaller) do this, including:
-
It breaks code inspection. This means IDEs (e.g., PyCharm), static code analyzers (e.g.,
pep8
,pylint
), and so on. Any IDE worth its salt will highlight pretty much every line of your script as an error; likewise, any static code analyzer will cough, croak, and choke on the dead kitten that is your uninterpretable script. - It prevents interpretation of code outside of BETSE. Ideally, BETSE scripts and code snippets should be interpretable as ordinary Python without BETSE-based interventions.
A saner alternative, therefore, might be to define a new high-level convenience submodule resembling Matplotlib's matplotlib.plot
API – say, betse.be
. The betse.be
submodule would:
- Perform the importations listed in the prior section.
- Initialize BETSE (e.g., call
ignition.init()
).
Externally leveraging BETSE in REPLs and scripts would then reduce to from betse import be
, preserving both terseness and interpretability.
Anything else? Clueless Joe knows not.
REPL – What's That Mean?
What exactly does the ambiguous term "REPL" mean? There are more than a few available REPLs, including (in descending order of personal preference):
- The third-party ptpython REPL, a frontend for the low-level third-party Python Prompt Toolkit. The former implements an improved superset of the IPython REPL named
ptipython
, offering multiline and ViM-oriented awesome. The latter is probably too nitty and/or gritty for our needs. Still, it is a thing. I have never used either. But I could, someday. Don't judge me. - The third-party IPython 3.x REPL (e.g., the
ipython3
command passed no arguments). - The third-party bpython REPL. It's no ptpython or IPython. But at least it's not Python.
- The stock Python 3.x REPL (e.g., the
python3
command passed no arguments). It is demonstrably bad. But it demonstrably exists.
Anything else? My Google-fu is as decrepit as it is lackluster.
Given this, the BETSE REPL could then be dynamically based on the first such REPL available on the current system – say, by algorithmically iterating through this list for the first importable package providing a REPL (beginning with ptpython and falling back to Python itself).
Usage – There's More Than One Way to Skin the Mewling Cat
How exactly should the BETSE REPL and scripting functionality be exposed to end users? @dglmoore's existing implementation adds an additional information theoretic-specific inform options:
subsection to the default YAML configuration file, which:
- Lists the relative or absolute paths of all non-interactive scripts to be run and the simulation phase(s) (e.g.,
init
,sim
) after which to run them. - Adds a boolean controlling whether or not the interactive REPL is dropped into immediately before running these scripts.
Both are noteworthy approaches. Especially because we currently have nothing.
It might be helpful, however, to contemplate a POSIX-style CLI API mimicking that of the Python interpreter itself, enabling seamless integration with shell environments everywhere without requiring manual edits of configuration files. Specifically, a new py
subcommand of the BETSE CLI supporting both non-interactive and interactive usage might be implemented. When run:
-
With arguments,
betse py
would run all passed arguments as BETSE-enabled non-interactive scripts. -
Without arguments,
betse py
would enter a BETSE-enabled interactive REPL.
To expose in-memory BETSE internals, betse py
would also accept options indicating exactly when the passed scripts are to be run or REPL to be dropped into. These options might resemble:
-
--after-seed=${sim_config_filename}
, indicating after the passed simulation is seeded. -
--after-init=${sim_config_filename}
, indicating after the passed simulation is initialized. -
--after-sim=${sim_config_filename}
, indicating after the passed simulation is run.
All three, only two, only one, or no such options could be passed. For example, at the command line:
# Run the BETSE REPL after initializing this simulation!
$ betse py --after-init=sim_config.yaml
[betse] Initializing simulation...
.
.
.
[betse] Initialized simulation.
[betse] Welcome to the <<BETSE 0.4.0 | CPython 3.4.3 | Gentoo Base System 2.2>> REPL.
Type "copyright", "credits", or "license" for more information.
be -> the "betse.be" convenience API
? -> Introduction and overview of the BETSE REPL's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: print(be.__version__)
0.4.0
Do you really want to exit ([y]/n)? y
# Run two BETSE scripts after running this simulation!
$ echo 'from betse import be\nprint("I am " + be.NAME + ".")' > muh_script.py
$ echo 'from betse import be\print("Who are " + be.AUTHORS + "?")' > dat_script.py
$ betse py --after-sim=sim_config.yaml muh_script.py dat_script.py
[betse] Running simulation...
.
.
.
[betse] Ran simulation.
I am BETSE.
Who are Alexis Pietak, Cecil Curry?
You may be thinking: "What's the benefit of running non-interactive scripts via betse py
rather than just python3
, then?" I am thinking this, too. Well, besides the obvious access to in-memory BETSE internals. Perhaps betse py
also:
- Supports
--dirty
and--isolate
CLI options to preserve and isolate changes between simulation runs (respectively). Indeed, the--isolate
option could be trivially implemented under POSIX-compatible platforms (read: not Windows) to fork the current Python process, which would have the noteworthy benefit of effectively parallelizing scripts. In this case, a subset of the passed scripts could be concurrently interpreted as forked subprocesses, which the kernel would then assign to appropriate (…hopefully idle) logical processors. - Implicitly catches and logs exceptions raised by erroneous scripts and skips to the next script rather than brutally halting execution altogether.
Both seem generally useful.
IPython (...I Mean, Jupyter) Notebook
Jupyter. If that isn't the lamest malapropism I've never heard, I don't know what is. We shall henceforth only refer to this thing as the IPython Notebook.
The IPython Notebook and its HTML5 REPL is a good, noble, and beautiful thing. BETSE should provide out-of-the-box integration with this REPL. I don't use this REPL terribly often (read: never), so I'm probably not the best kludger to kludge IPython Notebook integration together.
Any precision here would be useful. I have none.