Infrastructure supporting external API - Redmine #2585
This issue describes the major feature developments necessary to support development of external API facilities with appropriate abstraction and integration testing. With issues like #988 still open, it is still possible to describe necessary GROMACS infrastructure changes to support API development and the best known API functionality goals.
The following list summarizes feature requirements established during development of https://github.com/kassonlab/gmxapi
- MPI environment sharing
- Binding external objects with interfaces for external code to retain access to launched tasks (such as objects instantiated during thread-MPI launch)
- Accessible simulation stop signal
- Launch MD runner through library calls
- Extensible ForceProviders
- Restraint framework / extensible “pull” code
- Libgmxapi build target and public headers
- Libgmxapi doxygen documentation
- Manage filesystem input and output locations
- Interaction / synchronization with predetermined integer time steps rather than time value.
Packaging and project structure
The following notes may belong in a different issue, such as #2045, but are provided here for context.
Long term expectations consistent with this proposal:
- Its modules may have headers internal to them,
- Exposes a library API of deliberately selected components
- Its modules may expose components to that library API e.g. IForceProvider,
- That library API must be unit tested so it is known to work
- does not expose a user-facing API (ie no installed headers)
- Library API will be stable over the lifetime of a release branch (ie from about the time of the official release),
- Soversion bumps with each major release
- Library API will not be stable in master branch (but must pass its own tests in CI)
- Expects to be able to link itself dynamically
- Expects to be able to link itself statically (and its dependencies, if available)
- Linked as position-independent code
- Requires python v3 (patch from Pascal in gerrit, also releng needs an update too, probably)
Libgmxapi in GROMACS repo
While the exposed symbols and public headers are reconsidered and as code in libgromacs continues to undergo more compartmentalization and modernization, it is convenient to define a second library and set of public headers. For now, we defer the question of whether libgmxapi would ultimately be subsumed into libgromacs, but we propose that the gmxapi headers ultimately replace the current set of public headers.
- Builds with access to libgromacs library API headers
- installs the user-facing C headers for client code (such as what Python module builds against)
- “plugin” API. E.g. a stable framework(s) for implementing various sorts of potentials while hiding IForceProvider e.g. by exposing adapters to things like domain-distributed positions. May warrant different versioning semantics than API/ABI supporting user interfaces: probably more stable API; possibly less stable ABI.
- (Proposing) semantic versioning (standard guarantees starting somewhere between 0.1 and 1.0) https://semver.org/
- One name or separate name for tMPI / MPI? (suggest “no”)
- Plugin API only in C, but has supported method to expose bindings to the inserted code that can be used in Python scripts, i.e plugins are not written in pure Python
- Main (high-level workflow) API complete only in python, but will migrate to C. (We can accelerate the C version if there is a need)
- No plans to load C plugin modules natively (ie no dlopen) because this is able to be handled more portably at Python level (or some other user interface code), but we need to be able to receive and register factory functions to get objects from external code
Parts of this discussion may belong under #988
A GROMACS Python package like https://github.com/kassonlab/gmxapi is nominally beyond the scope of this issue. A few points from #2045 worth mentioning, though:
- serves to prove libgmxapi functionality; needs integration testing with libgromacs and libgmxapi
- Ultimately, it should be available as part of a GROMACS installation, but should it be in the same repository and/or CMake build environment?
- pybind11 source bundled with repo
- Provides both an implementation and an API specification.
- Researchers can be compatible with it without depending on it.
- Writers of GROMACS extension code are free to use other Python bindings frameworks.
- We provide tools and helpers, but C helpers use pybind11
A big picture of planned development is necessary even before Redmine Issues exist, so milestones are enumerated with feature ID tags. Dependencies are better illustrated in the accompanying chart. Each numbered feature (chart node) is expected to be from 1 to 10 Gerrit changes, generally 3 to 5.
Proposed development targets
Gmx1 (this issue) Design documentation strategy / project management plan
- this issue and a cluster of Redmine issues with subtasks should be good
- documentation and visual aids like the attached progress chart should probably be in the repository somewhere
#2586) Versioned libgmxapi target for build, install, headers, docsGmx2 (Issue
#2756)Gmx3 Integration testing (Issue
- Gmxapi interfaces should continue functioning with unchanged semantics for other GROMACS changes, or API level needs to be incremented according to semantic versioning.
- External projects need to be tested outside of the gromacs build tree to validate external interfaces of installation. Suggested external projects: Python package, sample_restraint, yet-to-be-written integration test suite.
- Tests should be clear about the API version they are testing, and we should test all versions that aren’t unsupported (though we need a policy in this regard) and we can note whether new API updates are backwards compatible.
- Forward-compatibility testing: we should at least know whether we are breaking old client code and include release notes, regardless of policy
- ABI compatibility testing? (should we test mismatched compilers and such?)
- Example code in documentation should be tested, if possible.
Gmx4 Library access to MD runner
- mdrun CLI program is an API client
Relates to #2229
#2587)Provide runner with context managerGmx5 (Issue
Gmx6 Extensible MDModules and ForceProviders
- ForceProviders obtained after tMPI threads have spawned.
- MDModules list extended at runtime during simulation launch.
- External code may be provided to the runner to instantiate or get a handle to a module.
- Expanded Context class can broker object binding by registering and holding factory functions for modules, as well as other resources that may be implemented differently in different environments.
- Somewhere in here, MDModules either need access to the integral timestep number or the ability to register call-backs or signals on a schedule.
Relates to #2590, #2574, #1972
Do MDModules live in a scope of tight association with an integrator? Do we need other concepts, like RunnerModules? Or subdivisions like MDForceModule, MDObserverModule, MDControlModule?
Gmx7 Binding API for higher-level client code
Gmx8 Binding API for plug-in ForceProviders
Ultimately tied to gmx5 and gmx24, but we can start stabilizing the external interfaces now. The external interfaces are for (a) user interface / workflow management code, and (b) MD extension code. We define a simple message-passing C structure along with PyCapsule name and semantics. An MD extension object can provide a factory method with which the MD Runner can get an IMDModules interface at simulation launch. The object pointed to may exist before and/or after the lifetime of the simulation. It must be understood that the IMDModule handle will be obtained on every rank. Design should consider future infrastructure and needs, but does not need to implement now. (expressing data dependencies and locality, negotiating parallelism, expressing periodicity) Short-term implementation may require workarounds for some of these, but the workaround can mostly be segregated from this issue’s resolution.
Relates to #2590
Gmx9 Headers and adapter classes for Restraint framework
Relates to #1972, #2590, https://github.com/kassonlab/sample\_restraint
Gmx10 MD signalling API
Relates to #2224
Gmx11 Replace MPI_COMM_WORLD with a Context resource
Gmx12 Runtime API for sharing / discovering hardware / parallelism resources
- Libgmxapi requests resources from libgromacs from the current node
- CUDA environment can be manipulated but we shouldn’t have to deal with that for a while
- Evolving task scheduling interfaces, expressing data locality
- Concepts of time and timestep
Gmx13 API for working directory, input/output targets?
Gmx14 Generalized pull groups / “generalized sites”
Christian Blau actively working on this from mid-July
Gmx15 API logging resource
Log “file” artifacts are produced through API, allowing extensibility and abstraction from filesystem dependence. Progress has already been made in this direction, but the logging resource could be more clearly owned by the client code (or a Context object owned or managed on behalf of the client code) rather than created and destroyed in, say, the Mdrunner.
Also relates to #2570
Gmx16 Exception handler firewall
currently the gmx binary has a commandline runner thing that catches the exceptions, reports an error and exits, but the API can and should do something else, because it plays the same role as the commandline runner
Gmx17 API status object
- Status type defines the interface for discovering operation success or failure, plus details.
- Consistent status object interface is portable across Python, C, and C
- Status object can be used to ferry information across API boundaries from exceptions thrown. Exceptions could be chained / status nested.
- What are concerns and solutions for memory allocation for status objects? Should objects own one or generate one on function return?
- Should the API (or Context) keep a Status singleton? A Status stack? Or should operations create ephemeral Status objects, or objects implementing a Status interface?
- Should the status object contain strings, reference strings mapped by enum, or defer textual messages to messaging and logging facilities?
Gmx18 Thinner test harness (for API client tests)
Gmx19 API manipulation of simulation input / output
(for better testing) - GlobalTopology class and IGlobalTopologyUser interface underway will help here, so that client changes to the global topology can ripple through to the modules because the ones that care have registered themselves at setup time
Gmx20 Accessible test data resources
Gmx21 Break up runner state into a hierarchy of RAII classes with API hooks
- break up mdrun program into clearly defined layers and phases
- CLI program parses various inputs in order to launch an Mdrunner object that is CLI-agnostic
- launching tMPI threads and other significant changes of state establish a sequence or hierarchy of invariants through RAII and/or State pattern.
- Sebastian Wingbermuhle working now on aspects of this for hybrid MC/MD (ref #2375, …)
Gmx22 API management of input objects
- Structure, topology
- Simulation state
- Simulation parameters
- Runtime parameters / execution environment
- Anything else?
Gmx23 Event hooks or signals
Event hooks or signals for
- time step number or delta / trajectory advancement
- input configuration
- input topology
- input state
- simulation parameters
- output data streams
Gmx24 API expression of MDOptions interfaces and embedded user documentation
Gmx25 Avoid sys::exit
Generally, replace std::exit (gmx_fatal)with exceptions
- Root out gmx_fatal, clearly define regular exit points and exception throwers
- API firewall should catch exceptions from gmx and convert to status objects for ABI compatibility. (gmx17)
- Clearly document regular and irregular shutdown behavior under MPI, tMPI, and generally, specifying responsibilities
- Create issue tickets for discovered missing exception safety, memory leaks, opportunities for RAII refactoring, and complicated protocols that should either be better documented or replaced with a clearer hierarchy (or sequence) of invariants
gmx26 API messaging resources
Abstraction for status messages, such as are currently printed to stdout or stderr
gmx28 set simulation parameters from API
Short term: mdrun CLI-like functionality to override other input is sufficient
Long term: sufficient API to update parameters between phases of simulation work
Implementation roadmap is probably
- Inject argv fields
- Write to input_rec or other structures
- Interact with MDOptions framework
gmx29 API access to grompp functionality
- Generate runnable input from user input
- United implementation for workflow API and utility functions (e.g. possibility of deferred execution / data transfer)
- Ultimately should not require writing output to (tpr) file
- File inputs ultimately should be generalized to API objects
gmx30 API access to GROMACS file manipulation and topology manipulation tools.
- United implementation for workflow API and utility functions (e.g. possibility of deferred execution / data transfer)
- Utility API should be sufficient to reimplement CLI tools
- I/O should ultimately be separate from algorithm; filesystem interaction optional
- Consider feature requirements of other projects such as MDAnalysis.
#2698) gmx31 Documentation integration.(Issue
Establish policies and layout for external (installed) API documentation, extension API for plug-in developers, and developer documentation for API and library implementation levels. Integrate with previous `webpage-sphinx` and `doxygen` targets and output.
There are definitely design points for consideration that are left out of this list merely because they are not essential to gmxapi functionality or because gmxapi doesn’t have strong dependence on the ultimate design choice. These topics include:
- Task scheduling framework
- Insertion points in the MD loop
- Encapsulation of integrator
Further downstream, this infrastructure is necessary to support new high level interfaces to GROMACS, but the discussion of such interfaces is deferred as much as possible to separate issues to streamline incorporation of the changes proposed here in less public / stable code.
Criteria for Completion
This issue is resolved when sufficient infrastructure is in place to support ongoing development of the other subprojects in issue #2045 against the GROMACS master branch in Gerrit. In particular, the proof-of-concept client code at https://github.com/kassonlab/gmxapi and https://github.com/kassonlab/sample\_restraint would not require a forked specialized copy of GROMACS.
(from redmine: issue id 2585, created on 2018-07-24 by eirrgang)
- Revision 9864b201 by Eric Irrgang on 2018-08-29T12:32:34Z:
Allow extensible MDModules and forceProviders. supports gmxapi milestone 6, described at #2585. MDModules::Impl gets a std::vector for (shared) ownership of objects providing the IMDModule interface. An add() method is added to the MDModules public member functions, but the binding protocols are separated into separate issues to allow minimal changes and varying dependencies on other pending changes. Relates to #1972, #2229, #2492, #2574, #2590. Refs #2623 Change-Id: Ibb16d1453003213a49622810ed8bad4ed4b06e2d
- Revision 6c9906e6 by Eric Irrgang on 2018-09-23T11:59:19Z:
Add gmxapi::MDModule. Provide basic framework for passing objects between object libraries. Describe protocol for managing object lifetimes and sharing ownership across API environments (such as Python, C, and C++) with minimal dependencies. Supports gmxapi milestone 7: Refs #2585 Change-Id: I206adb7341b3005dd56137e7c7df207a08dd41c2
- Revision 24e70bfa by Eric Irrgang on 2018-10-03T10:07:25Z:
Add test data for gmxapi. Build input file for a small system for basic gmxapi testing. These test files either come from the mdrun tests or from manipulation of that data. With access to test data and API tools to manipulate input configurations, these files can be moved or removed. Relates to gmxapi milestone 20 as described in issue #2585 Change-Id: I0eeb1d15120d3eb5309ec38c57e4e6f60097885a
- Revision 1b34724f by Eric Irrgang on 2018-10-11T09:57:19Z:
Add gmxapi::Session. Allow API client to launch simulation work in a provided Context. Supports running a simulation from a TPR file. Support gmxapi milestone 7 Refs #2585 Change-Id: Ie57962a739ae722b1ced8dd75f0037f07be3e1fa
- Revision 5327f266 by Eric Irrgang on 2018-10-11T10:23:11Z:
Add gmxapi::Workflow. Hold a description of simulation work to be performed. Work description is a dependency graph that allows a Context implementation to configure and launch an API Session to satisfy data flow constraints. Initial implementation just provides command-line type input for mdrun in the form of a TPR filename. A more complete implementation would look more like the "work specification" described in https://doi.org/10.1093/bioinformatics/bty484 For more discussion on a full implementation, refer to https://github.com/kassonlab/gmxapi/issues?q=milestone%3Agmxapi_workspec_0_2+ Refs #2585 Change-Id: I2c9bb99d88a11893201489757453ef25bc35a52
- Revision 6f025f6e by Mark Abraham on 2018-10-12T08:08:48Z:
Exclude gmxapi source files from include sorting For now, we are taking no care of such aspects, so the attributes instruct the include sorter to ignore matching files. The check-source target suppresses such files in other ways. Refs #2585 Change-Id: Ib84a89bfa7d5eb54a5a3585d65725fedf98fbab4
- Revision 0c6b7e5a by Eric Irrgang on 2018-10-12T11:58:44Z:
Add restraints framework. Extension code can implement IRestraintPotential and register to provide MD forces and energy. Client code is not necessarily expected to implement the interface directly, but likely with the help of template headers. Restraint functors should ultimately be retrieved or instantiated from factory functions in the API Context manager provided to the simulation, but in the current implementation, registered restraints are made available during launch through the MdrunnerBuilder. For sample extension code, see https://github.com/kassonlab/sample_restraint Supports gmxapi milestone 9, Refs #2585 Change-Id: I48d57570721223eacca957c9a0f7ae98163ab6ff
- Revision a888e947 by Eric Irrgang on 2018-10-12T16:50:29Z:
Add API for adding MDModules / Restraints. Add gmxapi::MDWorkSpec, which can be passed through MDHolder. It is a near-term solution until an API Context manager can be more fully realized. A future implementation should instead register factories for simulation support code in a Context that is passed from the client. This would resolve ambiguity about the purpose of gmxapi::MDWorkSpec versus the more general gmxapi::Workflow, but in the present form, MDWorkSpec is a public API class, while Workflow is still an implementation detail. Supports gmxapi milestone 7. Refs #2585 Change-Id: I41fb13646c41b0f6e7690a0cedabd17eb3c47433
- Revision 8bc1688d by Eric Irrgang on 2018-10-12T20:22:33Z:
Enable restraints facilities. Extend gmxapi to allow a client to attach code to an MD simulation through the restraints framework. Attach any registered restraints during session launch. Support gmxapi milestones 7 and 9, described in Refs #2585 Change-Id: I61f7b52f1490e7692cdf7f14c7f46f8970b1c30a
- Revision 7814d727 by Mark Abraham on 2018-10-15T08:41:05Z:
De-duplicating mdrun setup Made the API context setup use LegacyMdrunOptions to keep the two client setup procedures in step and avoid duplicating code. Refs #2585 Change-Id: I9909e26a41fd2015667c2f86335c09b6f9708e99
- Revision 7347b922 by Eric Irrgang on 2018-10-15T12:22:29Z:
Release notes for restraint module and gmxapi. Describe new features supporting gmxapi client code and runtime binding of restraint code. Add "gmxapi external API" section to install guide. Refs #2585 Change-Id: I3da0d130b07862abc90387ade781d111ccbcd740
- gmxapi milestone dependencies and status (graphical)
- chart.gv gmxapi milestone dependencies and status (DOT format text)