gmxapi integration testing - Redmine #2756
Supports gmxapi milestone 3 at parent issue.
Integration testing
GROMACS 2019 defines gmxapi 0.0.7. The kassonlab/gmxapi and kassonlab/sample_restraint repositories are the principal client software packages to test against a gmxapi compatible GROMACS installation. These packages are probably compatible with GROMACS 2020, but are not likely to be tested in the GROMACS project infrastructure.
GROMACS 2020 coincides with the definition of gmxapi 0.1, which is not fully compatible with GROMACS 2019, and probably will not be.
GROMACS infrastructure will include full gmxapi testing on GitLab Runner, but not on Jenkins.
Goals
- Gmxapi interfaces should continue functioning with unchanged semantics for other GROMACS changes, or API level needs to be incremented according to semantic versioning.
- External projects need to be tested outside of the gromacs build tree to validate external interfaces of installation. Suggested external projects: Python package, sample_restraint, validation test suite.
- Tests should be clear about the API version they are testing, and we should test all versions that aren’t unsupported (though we need a policy in this regard) and we can note whether new API updates are backwards compatible.
- Forward-compatibility testing: we should at least know whether we are breaking old client code and include release notes, regardless of policy
- ABI compatibility testing TBD. (should we test mismatched compilers and such?)
- Example code in documentation should be tested, where possible.
Matrix
Matrix axes contain the following.
- Python versions 3.5, 3.6, 3.7, 3.8
- MPI implementations mpich, mvapich, openmpi
- C compilers (all supported clang, gcc, intel…)
- GROMACS with MPI, thread-MPI, and no MPI
- GPU domain decomposition (relevant mostly for MD plugin testing, but possibly more)
- GROMACS floating point precision
- GROMACS branches master, release-2020, and beyond
- gmxapi 0.1, 0.2, and beyond
Versions tested
Nightly
For version information not specified in the matrix description, I think it is appropriate to test the most recent minor version or dependency version.
pre-commit
- Python patch version: default or most recent at time of configuration.
- Python package dependencies: minimum version described in package requirements.
- C dependencies (pybind, googletest, Eigen?): minimum version described in package requirements.
- cross verification (from master) with a release branch: oldest known compatible tag.
- cross verification (from release branch patch) with master: ti
Tasks
-
Define testing matrices for the various schedules (nightly, pre-commit, others?). Choose the sparse elements to test. -
Add “nightly-test” stage? -
Set up nightly rebuild of Python “venvs”s. Tagged artifacts? Some other caching mechanism? -
Generate initial pre-commit venvs (Keys: Python version, MPI version, and C toolchain.) -
Document update procedure for pre-commit venvs. -
Is there a GROMACS installation artifact already generated? -
What stage should client software be built and tested in? -
Extend .test-script-template for pytest with JUnit XML output.
Job dependencies
Apologies for the poor ASCII art…
build GROMACS -> install GROMACS -> build sample_restraint -> C++ unit tests
>-> pytest tests (MPI does not need as full coverage; could depend on other variables)
\-> install gmxapi -> non-MPI pytest tests
\-> MPI pytest tests
\-> acceptance tests(?)
The most expensive part here is the GROMACS build and install. The client software builds take a few seconds. The pytest unit tests take well under a minute. These timings may be comparable to the container launch or artifact handling, so it is probably sufficient to have a single stage to
- install gmxapi
- build (and install) sample_restraint
3-7. run the various tests.
The choice also depends whether there is a reason to have more stages versus wider stages, because we should plan to cross-check versions of the gmxapi Python package and versions of the sample code, at least until we are confident that we have well-characterized and acceptable compatibility guarantee and versioning API.
(from redmine: issue id 2756, created on 2018-11-14 by eirrgang)
- Relations:
- relates #2912 (closed)
- relates #3033 (closed)
- relates #3111 (closed)
- relates #3133 (closed)
- relates #3014 (closed)
- blocks #3271 (closed)
- blocks #3272 (closed)
- blocks #3408 (closed)
- parent #2585 (closed)
- Changesets:
- Revision 3fd14b5f by Paul Bauer on 2018-12-10T16:07:20Z:
Disable gmxapi by default
Due to outstanding issues with the integration testing and tests failing
with large number of ranks, the gmxapi default has been changed to not
be build. In Jenkins, all supported builds still are still set to build
with GMXAPI enabled.
Refs #2765, #2722, #2756
Change-Id: I2cc42c461edc206aaa30be6cac3db0a52ccae991
- Revision 4ed5d8edc187d3d36e62b4919740dbd2bb78f2b4 by Eric Irrgang on 2019-04-24T13:12:00Z:
Read GMXRC in gmxapi docker test environment.
Allows expected behavior for gmxapi/ci-... docker containers and ensures
that GROMACS components available during image build are available when
running.
Allows `gmx` binary to be easily discovered by Python wrappers.
Ref: #2756
Gerrit patch set 9390/4
- Revision 354da80619bc4c4f677c7e931cfea2cde559305d by Eric Irrgang on 2019-04-24T13:31:08Z:
More Python testing infrastructure for gmxapi.
* Add and improve test fixtures.
* Generate MD input files for tests.
Refs: #2756
Refs: #2896
Gerrit patch set 9392/2
- Revision 929343d1 by Eric Irrgang on 2019-05-09T11:12:45Z:
More Python testing infrastructure for gmxapi.
* Add and improve test fixtures.
* Generate MD input files for tests.
Refs: #2756
Refs: #2896
Change-Id: I15ac8e44844cbf699cfe362e2fa443d07a355b3d
- Revision b67801d7545d3c78747587a155a3474be39c7cb5 by Eric Irrgang on 2019-06-18T17:34:03Z:
Updates to gmxapi docker entry points.
* Read GMXRC in gmxapi docker test environment. Allows `gmx` binary to
be easily discovered by Python wrappers. Supports expected behavior
for gmxapi/ci-... docker containers and ensures that GROMACS
components available during image build are available when running.
* Allow more granular testing. Rename and reorganize entry points for
the docker test containers.
Ref: #2756
updates from new patch set to change 9390
- Revision e5a8e153 by Eric Irrgang on 2019-06-19T09:38:32Z:
Updates to gmxapi docker entry points.
* Read GMXRC in gmxapi docker test environment. Allows `gmx` binary to
be easily discovered by Python wrappers. Supports expected behavior
for gmxapi/ci-... docker containers and ensures that GROMACS
components available during image build are available when running.
* Allow more granular testing. Rename and reorganize entry points for
the docker test containers.
Ref: #2756
Change-Id: I5e781fae4709e4ccb718a8295c3fbe68dd59de44
- Revision 42adebaf by Eric Irrgang on 2019-07-24T04:19:22Z:
Improve gmxapi docker testing environment.
* Generalize venv directory.
* Add some usage notes.
* Better support alternative users in executing containers.
* Remove stale TODO.
* Allow for passing additional arguments to pytest.
Refs #2756
Change-Id: I9f67d46aa93fb789825cb78cc2a95b24f7dfccaa
- Revision 7cf28864 by Eric Irrgang on 2019-07-24T09:58:18Z:
Clarify use of Python venv in gmxapi docker images.
Refs #2756
Change-Id: I182c83be424fcf134bc0f3ce7c41a58c6fb1f07e
- Revision 6ec0f1f1 by Eric Irrgang on 2019-08-08T18:19:44Z:
Allow gmxapi Python package tests to be run from GROMACS build tree.
* Configure gmxapi._gmxapi to build with the ability to find libgmxapi.
* Add a gmxapi_pytest custom CMake target to invoke pytest.
Refs #2961
Refs #2756
Change-Id: I64ca82afbf37da3c37759ec302c2ac9cc1666de2
- Revision 4217d8fc by Eric Irrgang on 2019-08-21T13:37:30Z:
Enable GMXAPI by default.
Change default CMake configuration to enable the build and install of
libgmxapi and associated headers.
Refs #2756
Refs #3059
Change-Id: I5688a1ad6b524882090201014fc34cb34c1e3748