ENH: FileIONetCDF of local fields
The IO of local fields seems to be an ultimate performance killer of the FileIONetCDF. This has probably to do with its bad implementation resulting in single calls of ncmu_put_varm
see file_io_netcdf.cc 2556-2606 for each local pixel, originating from the complicated structure of local fields in the NetCDF file.
I recognized this performance problem when writing the fields of a 2D simulation with 4096x4096 grid points on 320cores.
fields written | # frames | file size (GB) | time (min) |
---|---|---|---|
stress + strain | 3 | ~6.1 | ~0.6 |
stress + strain + local (4096x4095) Youngs modulus | 3 | ~6.9 | ~87.8 |
Without having tested it, I would guess the FileIO performance gets even worse for less cores.
Edited by Richard Leute