CUPM: we should raise an error in VecCUDAGetArrayWrite() and friends with non-cuda vectors even --with--debugging=0
This is discussed in petsc-slack #general channel. With snes/tutorials/ex19 in debug mode,
$ ./ex19 -snes_monitor -ksp_converged_reason -dm_mat_type aijcusparse
lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
0 SNES Function norm 2.391552133017e-01
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: Object (seq) is not seqcuda or mpicuda
[0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!
[0]PETSC ERROR: Option left: name:-ksp_converged_reason (no value) source: command line
[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.18.2-302-gd4a1a345 GIT Date: 2022-12-07 13:38:02 +0000
[0]PETSC ERROR: ./ex19 on a arch-kokkos-dbg named petsc-gpu-01 by jczhang Fri Jan 6 09:31:26 2023
[0]PETSC ERROR: Configure options --PETSC_ARCH=arch-kokkos-dbg --with-debugging=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=0 --COPTFLAGS="-g -O0" --FOPTFLAGS="-g -O0" --CXXOPTFLAGS="-g -O0" --download-hdf5 --with-cuda --with-cudac=nvcc --download-kokkos --download-kokkos-kernels
[0]PETSC ERROR: #1 getarray() at /home/jczhang/petsc/include/petsc/private/veccupmimpl.h:804
[0]PETSC ERROR: #2 getarray() at /home/jczhang/petsc/include/petsc/private/veccupmimpl.h:853
[0]PETSC ERROR: #3 VecCUPMGetArrayAsync_Private() at /scratch/jczhang/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm.hpp:1623
[0]PETSC ERROR: #4 VecCUPMGetArrayWriteAsync() at /scratch/jczhang/petsc/src/vec/vec/impls/seq/cupm/cuda/../vecseqcupm.hpp:1672
[0]PETSC ERROR: #5 VecCUDAGetArrayWrite() at /scratch/jczhang/petsc/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.cu:281
[0]PETSC ERROR: #6 MatSolve_SeqAIJCUSPARSE_ILU0() at /scratch/jczhang/petsc/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu:1363
[0]PETSC ERROR: #7 MatSolve() at /scratch/jczhang/petsc/src/mat/interface/matrix.c:3602
[0]PETSC ERROR: #8 PCApply_ILU() at /scratch/jczhang/petsc/src/ksp/pc/impls/factor/ilu/ilu.c:198
[0]PETSC ERROR: #9 PCApply() at /scratch/jczhang/petsc/src/ksp/pc/interface/precon.c:441
[0]PETSC ERROR: #10 KSP_PCApply() at /home/jczhang/petsc/include/petsc/private/kspimpl.h:380
[0]PETSC ERROR: #11 KSPInitialResidual() at /scratch/jczhang/petsc/src/ksp/ksp/interface/itres.c:64
[0]PETSC ERROR: #12 KSPSolve_GMRES() at /scratch/jczhang/petsc/src/ksp/ksp/impls/gmres/gmres.c:227
[0]PETSC ERROR: #13 KSPSolve_Private() at /scratch/jczhang/petsc/src/ksp/ksp/interface/itfunc.c:898
[0]PETSC ERROR: #14 KSPSolve() at /scratch/jczhang/petsc/src/ksp/ksp/interface/itfunc.c:1070
[0]PETSC ERROR: #15 SNESSolve_NEWTONLS() at /scratch/jczhang/petsc/src/snes/impls/ls/ls.c:210
[0]PETSC ERROR: #16 SNESSolve() at /scratch/jczhang/petsc/src/snes/interface/snes.c:4693
[0]PETSC ERROR: #17 main() at ex19.c:152
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -dm_mat_type aijcusparse (source: command line)
[0]PETSC ERROR: -ksp_converged_reason (source: command line)
[0]PETSC ERROR: -malloc_dump (source: environment)
[0]PETSC ERROR: -snes_monitor (source: command line)
but in non-debug mode,
$ ./ex19 -snes_monitor -ksp_converged_reason -dm_mat_type aijcusparse
lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
0 SNES Function norm 2.391552133017e-01
Linear solve converged due to CONVERGED_ATOL iterations 0
Number of SNES iterations = 0
@prj- made some good points
PetscCheck() always aborts, even --with-debugging=0. The same is not true with PetscCheckTypeNames()
it should not be named PetscCheckTypeNames(), but something like PetscAssertTypeNames(), or whatever, just no Check.
Edited by Junchao Zhang