Skip to content

GitLab

Last edited by Oliver Perks
Page history

nwchem

URL: http://www.nwchem-sw.org

Categories: open-source, application,

NWChem: Open Source High-Performance Computational Chemistry. NWChem aims to provide its users with computational chemistry tools that are scalable both in their ability to treat large scientific computational chemistry problems efficiently, and in their use of available parallel computing resources from high-performance parallel supercomputers to conventional workstation clusters.

Versions

Build Details For Version 7.0 ACfL

NWChem 7.0 presents a number of improvements for building and running on Arm based systems such as:

  • Removal of deprecated MPI-1 interface calls
  • Auto recognition of Arm Compiler
  • No need to patch

A number of the QA tests still fail, most due to small precision errors - so validation is highly encouraged.

Configuration

  1. NWChem 7.0
  2. Arm Compiler for Linux 20.0
  3. Arm Perforamce Libraries 20.0
  4. Open MPI version 4.0.2
  5. Tested on TX2 running CentOS 8
  6. Last updated 14/05/20

Build instructions

Downloading and unpack the packages

wget https://github.com/nwchemgit/nwchem/archive/v7.0.0-release.tar.gz
tar xf v7.0.0-release.tar.gz
cd nwchem-7.0.0-release
export NWCHEM_TOP=$PWD

Modify the Make file

patch -p0 <<EOF
--- src/config/makefile.h   2020-05-12 07:32:42.500000000 -0500
+++ src/config/makefile.h.arm   2020-02-25 19:08:40.000000000 -0600
@@ -2273,6 +2273,14 @@
     endif
   endif
 
+  ifeq (\$(_CC),armclang)
+    COPTIONS   +=   -O3 -funroll-loops -mcpu=native -armpl
+    ifdef USE_OPENMP
+      COPTIONS += -fopenmp
+    endif
+  endif
+
+
   ifeq (\$(_FC),gfortran)
     ifdef  USE_GPROF
       FOPTIONS += -pg
@@ -2315,13 +2323,13 @@
 
     DEFINES   +=   -DARMFLANG
     LINK.f = \$(FC)  \$(LDFLAGS) 
-    FOPTIMIZE  = -O3 -Mfma -ffp-contract=fast
+    FOPTIMIZE  = -O3 -Mfma -ffp-contract=fast -fno-backslash
     ifeq (\$(V),1)
     \$(info     ARMFLANG FOPTIMIZE = \${FOPTIMIZE})
     endif
 
     FDEBUG += -g -O 
-    FOPTIMIZE +=  -mtune=native
+    FOPTIMIZE +=  -mcpu=native -armpl
 
     ifndef USE_FPE
       FOPTIMIZE  += -ffast-math #2nd time
EOF

Build configuration

export CC=armclang
export CXX=armclang++
export FC=armflang

export NWCHEM_TARGET=LINUX64
export ARMCI_NETWORK=MPI-PR

# Setup MPI PATHS
# Set to "y" to indicate that NWChem should be compiled with MPI
export USE_MPI=y
# Set to "y" for the NWPW module to use fortran-bindings of MPI (Generally set when USE_MPI is set)
export USE_MPIF=y
# Set to "y" for the NWPW module to use Integer*4 fortran-bindings of MPI. (Generally set when USE_MPI is set on most platforms)
export USE_MPIF4=y

# Use this to guess MPI value or set yourself
${NWCHEM_TOP}/src/tools/guess-mpidefs

# export MPI_INCLUDE="</path/to/mpi/includes>"
# export MPI_LIB="</path/to/mpi/libs/dir>"
# export LIBMPI="<list of mpi libraries>"


export NWCHEM_MODULES="all"
export BLAS_LIBS="-L$ARMPL_DIR/lib -larmpl_ilp64"
export BLASOPT=${BLAS_LIBS}
export BLAS_SIZE=8

# Same for LAPACK
# export LAPACK="-L</path/to/arm/performace/libraries/dir> -larmpl_ilp64"
# If you have used the provided module then you can just use
export LAPACK="-L$ARMPL_DIR/lib -larmpl_ilp64"
export LAPACK_LIBS="$LAPACK"
export LAPACK_LIB="$LAPACK"
export LAPACK_SIZE=8

# possibly need this for the configure step for GA ?
FFLAG_INT=-i8

Build and install

cd $NWCHEM_TOP/src
make nwchem_config
make -j 28 | tee make.log

Testing

# Run some QA tests
#--------------------
cd $NWCHEM_TOP/QA
export NWCHEM_EXECUTABLE=$NWCHEM_TOP/bin/${NWCHEM_TARGET}/nwchem
./doqmtests.mpi 16 | tee doqmtests.mpi.log

There are a number of failed test cases, see notes note at end.

Build Details For Version 6.8 Arm

Configuration

  1. NWChem 6.8
  2. Arm compiler version 18.0
  3. Arm Perforamce Libraries 18.0
  4. Open MPI version 2.1.2
  5. Tested on TX2 running Ubuntu 16.04
  6. Last updated 14/02/18

Build instructions

Downloading and unpack the packages

wget https://github.com/nwchemgit/nwchem/releases/download/v6.8-release/nwchem-6.8-release.revision-v6.8-47-gdf6c956-src.2017-12-14.tar.bz2

# Unpack tar file of src
tar jxf nwchem-6.8-release.revision-v6.8-47-gdf6c956-src.2017-12-14.tar.bz2

# set NWCHEM_TOP  to <your path>/nwchem-6.8
export NWCHEM_TOP=`pwd`/nwchem-6.8

Compiler configuration

export CC=armclang
export CXX=armclang++
export FC=armflang

Modify src

Replace the makefile.h in $NWCHEM_TOP/src/config with this or apply the following as a patch (from the $NWCHEM_TOP directory).

cd $NWCHEM_TOP
patch -p0 <<EOF
--- src/config/makefile.h   2018-02-02 10:30:28.399555033 +0000
+++ src/config/makefile.h.arm   2018-02-02 10:29:47.485239331 +0000
@@ -1592,6 +1592,9 @@
       ifeq (\$(CC),icc)
         _CC=icc
       endif
+     ifeq (\$(CC),armclang)
+       _CC=armclang
+     endif
       ifeq (\$(FC),pgf90)
         _FC=pgf90
       endif
@@ -1613,6 +1616,9 @@
          ifeq (\$(FC),xlf)
            _FC=xlf
          endif
+     ifeq (\$(FC),armflang)
+      _FC=armflang
+     endif
       ifndef _FC
       FC=gfortran
       _FC=gfortran
@@ -2141,6 +2147,92 @@
       endif
 endif
 
+     ifeq (\$(_CPU),\$(findstring \$(_CPU),aarch64))
+ 
+       ifdef  USE_DEBUG
+         FOPTIONS += -g
+         COPTIONS += -g
+         LDOPTIONS += -g
+         LDFLAGS += -g
+       endif
+  
+       ifeq (\$(_CC),gcc)
+         COPTIONS   +=   -O3 -funroll-loops -ffast-math 
+         ifdef USE_OPENMP
+           COPTIONS += -fopenmp
+         endif
+       endif
+ 
+       ifeq (\$(_FC),gfortran)
+ #gcc version 4.1.0 20050525 (experimental)
+         ifdef  USE_GPROF
+           FOPTIONS += -pg
+           COPTIONS += -pg
+           LDOPTIONS += -pg
+           LDFLAGS += -pg
+         endif
+         LINK.f = \$(FC)  \$(LDFLAGS) 
+         FOPTIMIZE  += -O3 
+         ifeq (\$(GNU_GE_6),true)
+           FOPTIMIZE += -fno-tree-dominator-opts # solvation/hnd_cosmo_lib breaks
+           FOPTIONS += -fno-tree-dominator-opts # solvation/hnd_cosmo_lib breaks
+           FDEBUG += -fno-tree-dominator-opts # solvation/hnd_cosmo_lib breaks
+         endif
+  
+         FOPTIMIZE  += -fprefetch-loop-arrays #-ftree-loop-linear
+         ifeq (\$(GNU_GE_4_8),true)
+           FOPTIMIZE  += -ftree-vectorize   -fopt-info-vec
+         endif
+
+         FDEBUG += -g -O 
+         ifdef USE_F2C
+ #possible segv with use of zdotc (e.g. with GOTO BLAS)
+ #http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20178
+          FOPTIONS +=  -ff2c -fno-second-underscore
+         endif
+         ifeq (\$(GNU_GE_4_6),true) 
+           FOPTIMIZE +=  -mtune=native
+           FOPTIONS += -finline-functions
+         endif
+         ifndef USE_FPE
+           FOPTIMIZE  += -ffast-math #2nd time
+         endif
+         ifdef  USE_FPE
+           FOPTIONS += -ffpe-trap=invalid,zero,overflow  -fbacktrace
+         endif
+       endif  # end of gfortran
+ 
+       ifeq (\$(_FC),armflang)
+ 
+         ifdef USE_SHARED
+           FOPTIONS+= -fPIC
+         endif
+ 
+         DEFINES   +=   -DARMFLANG
+ 	    LINK.f = \$(FC)  \$(LDFLAGS) 
+         #FOPTIMIZE  += -O3 
+         FOPTIMIZE  = -O3 -Mfma -ffp-contract=fast
+         # not in armflang
+         # FOPTIMIZE  += -fprefetch-loop-arrays #-ftree-loop-linear
+         ifeq (\$(GNU_GE_4_8),true)
+         # not in armflang
+           FOPTIMIZE  += -ftree-vectorize   -fopt-info-vec
+         endif
+ 
+         FDEBUG += -g -O 
+         FOPTIMIZE +=  -mtune=native
+         # not in armflang
+         # FOPTIONS += -finline-functions
+  
+         ifndef USE_FPE
+           FOPTIMIZE  += -ffast-math #2nd time
+         endif
+         ifdef  USE_FPE
+           FOPTIONS += -ffpe-trap=invalid,zero,overflow  -fbacktrace
+         endif
+       endif
+     endif # end of aarch64
+
 ifeq (\$(_CPU),\$(findstring \$(_CPU), ppc64 ppc64le))
 # Tested on Red Hat Enterprise Linux AS release 3 (Taroon Update 3)
 # Tested on SLES 9
EOF

Modify src/inp/inp.F

sed -i 's/#if defined(DECOSF)/#if defined(ARMFLANG) || defined(DECOSF)/' $NWCHEM_TOP/src/inp/inp.F

Modify src/optim/string/string.F

sed -i 's/logical freeze1,freezeN,projection1/logical freeze1,freezeN,projection1,tmpVar/' $NWCHEM_TOP/src/optim/string/string.F
sed -i '/if ( .not. (task_gradient(rtdb).or.ignore))/a\            tmpVar = task_gradient(rtdb)' $NWCHEM_TOP/src/optim/string/string.F
sed -i '/tmpVar = task_gradient(rtdb)/a\            if ( .not. (tmpVar.or.ignore))' $NWCHEM_TOP/src/optim/string/string.F                       
sed -i 's/            if ( .not. (task_gradient(rtdb).or.ignore))/*            if ( .not. (task_gradient(rtdb).or.ignore))/' $NWCHEM_TOP/src/optim/string/string.F

Build configuration

cd $NWCHEM_TOP

# Set up NWChem env vars
# $NWCHEM_TARGET defines your target platform, e.g.
export NWCHEM_TARGET=LINUX64

# ARMCI_NETWORK must be defined in order to achieve best performance on high performance networks, e.g.
export ARMCI_NETWORK=MPI-PR

# Setup MPI PATHS
# Set to "y" to indicate that NWChem should be compiled with MPI
export USE_MPI=y
# Set to "y" for the NWPW module to use fortran-bindings of MPI (Generally set when USE_MPI is set)
export USE_MPIF=y
# Set to "y" for the NWPW module to use Integer*4 fortran-bindings of MPI. (Generally set when USE_MPI is set on most platforms)
export USE_MPIF4=y

# You can try to run ${NWCHEM_TOP}/src/tools/guess-mpidefs to guess the values for these MPI defs
export MPI_INCLUDE="</path/to/mpi/includes>"
export MPI_LIB="</path/to/mpi/libs/dir>"
export LIBMPI="<list of mpi libraries>"

# NWCHEM_MODULES defines the modules to be compiled
export NWCHEM_MODULES="all"

# Optimized armpl math libraries - use ilp64 since NWChem uses 64 bit integers on a 64bit systems
# export BLAS_LIBS="-L</path/to/arm/performace/libraries/dir> -larmpl_ilp64"
# If you have used the provided module then you can just use
export BLAS_LIBS="-L$ARMPL_DIR/lib -larmpl_ilp64"
export BLASOPT=${BLAS_LIBS}
export BLAS_SIZE=8

# Same for LAPACK
# export LAPACK="-L</path/to/arm/performace/libraries/dir> -larmpl_ilp64"
# If you have used the provided module then you can just use
export LAPACK="-L$ARMPL_DIR/lib -larmpl_ilp64"
export LAPACK_LIBS="$LAPACK"
export LAPACK_LIB="$LAPACK"
export LAPACK_SIZE=8

# possibly need this for the configure step for GA ?
FFLAG_INT=-i8

Build and install

cd $NWCHEM_TOP/src
make nwchem_config
make -j 28 | tee make.log

Testing

# Run some QA tests
#--------------------
cd $NWCHEM_TOP/QA
export NWCHEM_EXECUTABLE=$NWCHEM_TOP/bin/${NWCHEM_TARGET}/nwchem
./doqmtests.mpi 16 | tee doqmtests.mpi.log

There are a number of failed test cases, see notes note at end.

Build Details For Version 6.8 GCC

Configuration

  1. NWChem 6.8
  2. GCC compiler version 7.1
  3. Arm Perforamce Libraries 18.0
  4. Open MPI version 2.1.2
  5. Tested on TX2 running Ubuntu 16.04
  6. Last updated 31/01/18

Build instructions

Downloading and unpack the packages

wget https://github.com/nwchemgit/nwchem/releases/download/v6.8-release/nwchem-6.8-release.revision-v6.8-47-gdf6c956-src.2017-12-14.tar.bz2

# Unpack tar file of src
tar jxf nwchem-6.8-release.revision-v6.8-47-gdf6c956-src.2017-12-14.tar.bz2

# set NWCHEM_TOP  to <your path>/nwchem-6.8
export NWCHEM_TOP=`pwd`/nwchem-6.8

Compiler configuration

export CC=gcc
export CXX=g++
export FC=gfortran

Modify src

Replace the makefile.h in $NWCHEM_TOP/src/config with this or apply the following as a patch (from the $NWCHEM_TOP directory).

cd $NWCHEM_TOP
patch -p0 <<EOF
--- src/config/makefile.h   2018-02-02 10:30:28.399555033 +0000
+++ src/config/makefile.h.arm   2018-02-02 10:29:47.485239331 +0000
@@ -1592,6 +1592,9 @@
       ifeq (\$(CC),icc)
         _CC=icc
       endif
+     ifeq (\$(CC),armclang)
+       _CC=armclang
+     endif
       ifeq (\$(FC),pgf90)
         _FC=pgf90
       endif
@@ -1613,6 +1616,9 @@
          ifeq (\$(FC),xlf)
            _FC=xlf
          endif
+     ifeq (\$(FC),armflang)
+      _FC=armflang
+     endif
       ifndef _FC
       FC=gfortran
       _FC=gfortran
@@ -2141,6 +2147,92 @@
       endif
 endif
 
+     ifeq (\$(_CPU),\$(findstring \$(_CPU),aarch64))
+ 
+       ifdef  USE_DEBUG
+         FOPTIONS += -g
+         COPTIONS += -g
+         LDOPTIONS += -g
+         LDFLAGS += -g
+       endif
+  
+       ifeq (\$(_CC),gcc)
+         COPTIONS   +=   -O3 -funroll-loops -ffast-math 
+         ifdef USE_OPENMP
+           COPTIONS += -fopenmp
+         endif
+       endif
+ 
+       ifeq (\$(_FC),gfortran)
+ #gcc version 4.1.0 20050525 (experimental)
+         ifdef  USE_GPROF
+           FOPTIONS += -pg
+           COPTIONS += -pg
+           LDOPTIONS += -pg
+           LDFLAGS += -pg
+         endif
+         LINK.f = \$(FC)  \$(LDFLAGS) 
+         FOPTIMIZE  += -O3 
+         ifeq (\$(GNU_GE_6),true)
+           FOPTIMIZE += -fno-tree-dominator-opts # solvation/hnd_cosmo_lib breaks
+           FOPTIONS += -fno-tree-dominator-opts # solvation/hnd_cosmo_lib breaks
+           FDEBUG += -fno-tree-dominator-opts # solvation/hnd_cosmo_lib breaks
+         endif
+  
+         FOPTIMIZE  += -fprefetch-loop-arrays #-ftree-loop-linear
+         ifeq (\$(GNU_GE_4_8),true)
+           FOPTIMIZE  += -ftree-vectorize   -fopt-info-vec
+         endif
+
+         FDEBUG += -g -O 
+         ifdef USE_F2C
+ #possible segv with use of zdotc (e.g. with GOTO BLAS)
+ #http://gcc.gnu.org/bugzilla/show_bug.cgi?id=20178
+          FOPTIONS +=  -ff2c -fno-second-underscore
+         endif
+         ifeq (\$(GNU_GE_4_6),true) 
+           FOPTIMIZE +=  -mtune=native
+           FOPTIONS += -finline-functions
+         endif
+         ifndef USE_FPE
+           FOPTIMIZE  += -ffast-math #2nd time
+         endif
+         ifdef  USE_FPE
+           FOPTIONS += -ffpe-trap=invalid,zero,overflow  -fbacktrace
+         endif
+       endif  # end of gfortran
+ 
+       ifeq (\$(_FC),armflang)
+ 
+         ifdef USE_SHARED
+           FOPTIONS+= -fPIC
+         endif
+ 
+         DEFINES   +=   -DARMFLANG
+ 	    LINK.f = \$(FC)  \$(LDFLAGS) 
+         #FOPTIMIZE  += -O3 
+         FOPTIMIZE  = -O3 -Mfma -ffp-contract=fast
+         # not in armflang
+         # FOPTIMIZE  += -fprefetch-loop-arrays #-ftree-loop-linear
+         ifeq (\$(GNU_GE_4_8),true)
+         # not in armflang
+           FOPTIMIZE  += -ftree-vectorize   -fopt-info-vec
+         endif
+ 
+         FDEBUG += -g -O 
+         FOPTIMIZE +=  -mtune=native
+         # not in armflang
+         # FOPTIONS += -finline-functions
+  
+         ifndef USE_FPE
+           FOPTIMIZE  += -ffast-math #2nd time
+         endif
+         ifdef  USE_FPE
+           FOPTIONS += -ffpe-trap=invalid,zero,overflow  -fbacktrace
+         endif
+       endif
+     endif # end of aarch64
+
 ifeq (\$(_CPU),\$(findstring \$(_CPU), ppc64 ppc64le))
 # Tested on Red Hat Enterprise Linux AS release 3 (Taroon Update 3)
 # Tested on SLES 9
EOF

Build configuration

cd $NWCHEM_TOP

# Set up NWChem env vars
# $NWCHEM_TARGET defines your target platform, e.g.
export NWCHEM_TARGET=LINUX64

# ARMCI_NETWORK must be defined in order to achieve best performance on high performance networks, e.g.
export ARMCI_NETWORK=MPI-PR

# Setup MPI PATHS
# Set to "y" to indicate that NWChem should be compiled with MPI
export USE_MPI=y
# Set to "y" for the NWPW module to use fortran-bindings of MPI (Generally set when USE_MPI is set)
export USE_MPIF=y
# Set to "y" for the NWPW module to use Integer*4 fortran-bindings of MPI. (Generally set when USE_MPI is set on most platforms)
export USE_MPIF4=y

# You can try to run ${NWCHEM_TOP}/src/tools/guess-mpidefs to guess the values for these MPI defs
export MPI_INCLUDE="</path/to/mpi/includes>"
export MPI_LIB="</path/to/mpi/libs/dir>"
export LIBMPI="<list of mpi libraries>"

# NWCHEM_MODULES defines the modules to be compiled
export NWCHEM_MODULES="all"

# Optimized armpl math libraries - use ilp64 since NWChem uses 64 bit integers on a 64bit systems
# export BLAS_LIBS="-L</path/to/arm/performace/libraries/dir> -larmpl_ilp64"
# If you have used the provided module to configure armpl then you can just use
export BLAS_LIBS="-L$ARMPL_DIR/lib -larmpl_ilp64"
export BLASOPT=${BLAS_LIBS}
export BLAS_SIZE=8

# Same for LAPACK
# export LAPACK="-L</path/to/arm/performace/libraries/dir> -larmpl_ilp64"
# If you have used the provided module to configure armpl then you can just use
export LAPACK="-L$ARMPL_DIR/lib -larmpl_ilp64"
export LAPACK_LIBS="$LAPACK"
export LAPACK_LIB="$LAPACK"
export LAPACK_SIZE=8

Build and install

cd $NWCHEM_TOP/src
make nwchem_config
make -j 28 | tee make.log

Testing

# Run some QA tests
#--------------------
cd $NWCHEM_TOP/QA
export NWCHEM_EXECUTABLE=$NWCHEM_TOP/bin/${NWCHEM_TARGET}/nwchem
./doqmtests.mpi 16 | tee doqmtests.mpi.log

There are a number of failed test cases, see notes note at end.

Testing

Testing of NWChem 6.8 with both gcc and Arm compiler. The test script doqmtests.mpi has been used to test these builds on a TX2 system. Currently there are a number of failed tests (currently 44 from 240 tests) but mostly for very minor numerical differences compared to the provided test result.

The following test cases from doqmtests.mpi fail for very minor numerical differences and are deemed acceptable. Ar_scs-mp2_cc-vqz, He_scs-mp2_cc-vqz, Ne_scs-mp2_cc-vqz, acr_lcblyp, bf_cis, ch3f-lc-wpbe, ch3f-lc-wpbeh, ch3f_zora_shielding, ch3radical_rot, ch3radical_unrot, ch4cl_zts, cho_bp_props, cytosine_ccsd, dft_s12gh, dft_xdm1, etf_hcons, h2o_bnl, h2o-camb3lyp-pol-cdfit, h2o_raman_3, h2o_raman_4, h2o_vscf, hess_biph, hess_ch4, hess_nh3_ub3lyp, hi_zora_sf, mep-test, mp2_si2h6, neb-fch3cl, neb-isobutene, o2_bnl, o2_zora_so, oh2, oniom4, prop_cg_nh3_b3lyp, spin_ecps_o_hess, tce_cr_eom_t_ch_rohf, tce_cr_eom_t_ozone, tce_eomsd_eomsol1, tce_eomsd_eomsol2, uoverlap.