Make sure Colvars interoperates stably and smoothly with GROMACS - Redmine #3357

added Task + 1 deleted label

Archive from user: Hubert Santuz

Hi,

I’m starting to re-write our interface Colvars-Gromacs to fit into the new MDModule machinery and I have a couple interrogations.
Colvars works by reading a separate configuration file, so 1) we don’t need to modify the mdp file but 2)we need to add a new input file to mdrun.

About 1), I have implemented a class herited from IMDModule :

    class Colvars final : public IMDModule
    {
    ...
    }

but from I understand, I still need to implement mdpOptionProvider() (due to the virtual declaration in IMDModule) even though I don’t need to modify the mdp options, right?

About 2), currently, it’s not possible to add a custom parameters to mdrun for a new file (for example gmx mdrun -colvars file.dat). We still have to modify legacymdrunoptions.h to hardcode the keyword?
Secondly, the only class we can use is IMDOutputProvider, there is no IMDInputProvider. Is this something you consider to add?

I’m not sure I’m posting to the right place so feel free to point me to a better resource if needed.

(from redmine: written on 2020-02-05 by gmxdefault)

Hello Hubert, thanks for the feedback.
I think you can have a look at the code Christian Blau wrote for the density fitting to see how to read in files during preprocessing.
We want to move away from people having to declare extra files at the invocation of mdrun (as it means that we can’t have the whole simulation defined in the tpr, affecting portability to different systems).

This means that the colvars information should also be part of the TPR in the end.

(from redmine: written on 2020-02-05 by acmnpv)

Archive from user: Giacomo Fiorin

Hello Paul, thanks for clarifying this important detail. This is a reasonable choice.

However, it would be very tricky to input all Colvars configuration parameters as individual MDP parameters for two reasons: (1) they follow a different format with its own parser, and (2) there are many Colvars keywords, not all of which are used very often and they would needlessly bloat the MDP doc page at http://manual.gromacs.org/current/user-guide/mdp-options.html.

Do you have an example that Hubert can use for embedding a text file into the TPR, consistent with the architecture of future GROMACS releases? From a quick look, all modules that apply external forces seem to use MDP parameters, with the exceptions of:

AWH (uses numbered awhinit?.xvg files as an alternative to the -awh command line option, which may be phased out at some point if I understand correctly).
Density fitting, which uses MDP parameters alongside a filename for the density file itself, which is probably too large to be embedded in the TPR (please correct me if appropriate).

Would a MDP parameter along the lines of colvars-configuration-filename work?

(from redmine: written on 2020-02-05 by gmxdefault)

I added Christian Blau to the discussion, so I can clarify things about the modules.

(from redmine: written on 2020-02-07 by acmnpv)

Archive from user: Erik Lindahl

We will gradually move towards hierarchical input files (and replacing mdp text files with YAML at some point).

However, user inputs should be parsed when preparing the input (i.e., when running grompp) - we don’t want to introduce a second-stage text file parsing that only happens at runtime.

(from redmine: written on 2020-02-27 by gmxdefault)

When the MdModules are called during grompp-time, they get a chance to do the file parsing when they receive the callback to store their internal parameters.

I suggest colvars subscribes to the preProcessingNotifications_ KeyValueTreeObjectBuilder that allows colvars to parse it’s inputs and stores it as “internal” parameters in a key value tree that then may be read from the module
signing up to the simulationSetupNotifications_ KeyValueTreeObject.

The easiest way is just dumping the complete input file as a string in a single KVT tree entry; then the .tpr would work as a plain container that just stores the colvars input file.

(from redmine: written on 2020-03-04 by cblau)

We are looking at the current master branch, and one thing we see is a change in how PBCs are handled in checkpoints. Checkpointed coordinates used to be unwrapped until at least 2020.3, and now they seem to be wrapped. That breaks the logic that Colvars uses, which is to rely on initial unwrapped coordinates that are fed to communicate_group_positions, which subsequently keeps track of shifts for us. So my question is: is this by design, and if yes, how can we adapt Colvars to find the info about unwrapped initial coordinates?

@cblau can you have a look at this?

Hey @cblau & @acmnpv. We've set aside the master branch for now and had a more detailed look at 2020.3. There, we don't have the same wrapping issue, but there is a global translation of the coordinates when they are not centered in the given input. What mdrun does is not strictly a centering though, but seems to be the closest thing to centering that can be obtained via global translation by an integer number of cell basis vectors. This is not as bad as the wrapping of individual atoms, but it could still be a problem if collective variables happen to depend on absolute Cartesian coordinates (unusual but possible). Ideally we'd need access to the unadulterated input coordinates.

I'm looking into this and see why we did that change. The sfhit-vectors should still give you the right way to reconstruct the actual coordinates, and coordinate_grouo_positions should work nevertheless. I'll check more thoroughly this Friday and let you know.

assigned to @acmnpv

assigned to @cblau

@cblau Note that the GitHub PR referenced at the top of this issue is now merged.

To summarize that conversation, we agreed that Hubert and @HENINJ will continue discussing directly in this GitLab issue any changes needed to interface with the MdModule framework and in general with the more current versions of GROMACS.

@giacomofiorin - that's perfect for us.

Upon some design discussions with @eirrgang, we lean towards

restructuring MDModules to become less monolithic by requiring fewer interface implementations (removing IOutputProvider, etc..) and provide rather notifications
enabling adding external modules at an early stage when building the runner

Hope that will make things easier for you as well.

assigned to @eirrgang

Hi,

As @giacomofiorin said, we finished the work on the Gromacs-Colvars interface. It's fully compatible and has support of serial/MPI/t-MPI, restart (through checkpoint), virial computations. The downside is a lot of Gromacs source files modified.

So, I started to work on an interface through MdModules, you can see a WIP here : https://gitlab.com/HubLot/gromacs/tree/colvars-interface. It's mainly inspired from densityFitting. I managed to have a minimalist working version but without MPI or restarting support for now.

I put the colvars MdModule files here src/gromacs/applied_forces/colvars and the colvars source files here src/gromacs/colvars.

Here some feedback about the MdModule and questions/suggestions to go further in the implementation:

It was a bit hard to have an overview of all required interfaces to implement, how they interact to each other and what's the purpose of each virtual method to implement or when they are called. I'm no C++ expert so maybe it doesn't help either :) The densityFitting code is great for understanding this a bit more. But for example, I still don't know how to use IMDOutputProvider interface (I saw it will disappear).

For colvars, I created 3 mdp options : colvars-active, colvars-filename, colvars-filename-restart implemented in ColvarsOptions with IMdpOptionProvider. Can you explain the purpose of initMdpTransform(...)? I managed to implement it with the help of densityFitting but I don't see where it's used.

The core of the interface is in ColvarsForceProvider which inherits both from IForceProvider and colvarproxy (a virtual class that link colvar and the MD engine).

I noticed you use a PrivateImplPointer<Impl> for the IForceProvider in DensityFitting. What is the purposed of it? (1) I didn't use it in Colvars but I can make the changes if needed.
To construct the class, I uses parameters retrieve from notifications through the class ColvarsSimulationsParameters. I modified MdModulesNotifier to retrieve a gmx_mtop_t object needed by colvars to access the mass and charges of the colvars atoms. But colvars will need also a temperature and the initial step when initialized, should I modified again the MdModulesNotifier? (2)
Colvars also need the initial coordinates of the colvars atoms to properly initialize the vector of PBC whole positions (xa_old), how I can achieve that ? (3) And if it's not done, I cannot use properly communicate_group_positions for MPI support.
In fact, the main issue is that the biased atoms are defined in the colvars filename which is parsed during construction of the ColvarsForceProvider() (due to the inheritance of colvarproxy), that's why also I passed a LocalAtomSetManager to the constructor and not a LocalAtomSet.
In calculateForces(), Colvars computes its forces only on the master so we need to assemble the colvars atoms through communicate_group_positions but the parameter gmx_bool bNS needed is not available from the ForceProviderInput. Is it possible to add it ? (4)

Finally, Colvars write is own output through a method write_output_files(). We used to called it at the end of simulation in the runner.cpp but here I don't know. (5)

This reply is already long so I'll stop here! I hope I've made myself clear. I've numbered my questions for easy follow up.

Thanks

(1) is easy - it adds a small useful thing to std::unique_ptr but is already proposed for simplification at #3684 (closed)

PrivateImplPointer<Impl> is now gone - just use std::unique_ptr

Thanks for the comprehensive update! I think this will be a priority for implementation next year, and we should see from the core team that this goes smoothly. I think we will discuss this internally and hopefully get back to you soon.

great, thanks for your responses.

marked this issue as related to #3283 (closed)

mentioned in issue #3283 (closed)

Following our recent discussions, you can find in the following document of how Colvars integrate with GROMACS. There is a comparison between the current MdModule implementation and the functional one (without the use of MdModule). There is also a 'pseudo' workflow on how colvars interface with GROMACS, which data is needed , what is done and where in the codebase/MdModule it is done. Hopefully, this will help non-colvars people to have a better comprehension of what we tried to achieve.

In the meantime, I also updated my fork (https://gitlab.com/HubLot/gromacs/tree/colvars-interface). I created small commits to spot more easily the changes. I'll use it as a base to propose small MRs

Make sure Colvars interoperates stably and smoothly with GROMACS - Redmine #3357

Designs

Child items ...

Activity

Make sure Colvars interoperates stably and smoothly with GROMACS - Redmine #3357

Relates to

Activity