Question: where to put and when to report GPU memory?
GPU memory can be easily and accurately estimated with CUDA Fortran on NVIDIA GPUs, here's a draft
SUBROUTINE print_gpu_mem()
#if defined(__CUDA)
use cudafor
USE util_param, ONLY : DP, stdout
implicit none
INTEGER(KIND=8) :: freeMem,totalMem
REAL(DP) :: rFreeMem,rUsedMem,rTotalMem
integer :: istat
istat=CudaMemGetInfo(freeMem,totalMem)
rTotalMem = totalMem/(10.**6)
rFreeMem = freeMem/(10.**6);
rUsedMem = (totalMem-freeMem)/(10.**6);
write(stdout,"(A20,F7.1,A3,F7.1,A3,F7.1,A8)") " GPU memory used/free/total: ",rUsedMem," ",rFreeMem, " ",rTotalMem
#endif
END SUBROUTINE
Since lack of memory is one of the mail source of problems when running the GPU version, I would like to tell the user something about the memory usage from time to time.
This could be reported in case of errors, but it's probably useful to report mem levels also during SCF. One reason to do this is because you may realize after a few SCF cycles that there won't be enough memory for derivatives in force evaluation planned for the end of the simulation.
Do you agree? How would you suggest to proceed? I think the code should enter UtilXlib (or devXlib) but I'm unsure about where to put the CALL print_gpu_mem
.
Thanks, Pietro
PS: yes, I will add MPI before any merge request, so that it reports min, max and avg available memory on all MPI tasks.