-baseline-averaging broken with -apply-facet-solutions
Baseline based averaging (via -baseline-averaging) fails in combinations with -apply-facet-solutions. The code runs fine up to the point of making the PSF. After that WSClean runs out of all(!) the available 512 GB RAM of the node (I suspect this is related to applying the h5parm solutions).
If I repeat exactly the same command, but taking out the -apply-facet-solutions, the memory usage is only about 44 GB at a maximum at the "== Constructing image ==" stage. So far away from crossing the 512 GB RAM limit of the node. The code finishes and the output images looks as I would expect (when not applying the h5parm solutions).
2022-Jun-02 20:44:00.960461 Writing psf image... DONE
2022-Jun-02 20:44:06.018110 == Constructing primary beam ==
2022-Jun-02 20:44:23.957316 Opening reordered part 0 spw 0 for /net/rijn2/data2/rvweeren/ELAIS_store/sub6asec_L686962_SB001_uv_12CFFDDA8t_140MHz.ms.sub.shift.avg.apply_infield.avg4ch.ms
2022-Jun-02 20:50:27.018273 == Constructing image ==
2022-Jun-02 20:50:27.018420 == Constructing image ==
2022-Jun-02 20:50:27.018455 == Constructing image ==
2022-Jun-02 20:50:27.018514 == Constructing image ==
2022-Jun-02 20:50:27.018556 == Constructing image ==
2022-Jun-02 20:50:27.018584 Opening reordered part 0 spw 0 for /net/rijn2/data2/rvweeren/ELAIS_store/sub6asec_L686962_SB001_uv_12CFFDDA8t_140MHz.ms.sub.shift.avg.apply_infield.avg4ch.ms
2022-Jun-02 20:50:27.018623 Opening reordered part 0 spw 0 for /net/rijn2/data2/rvweeren/ELAIS_store/sub6asec_L686962_SB001_uv_12CFFDDA8t_140MHz.ms.sub.shift.avg.apply_infield.avg4ch.ms
2022-Jun-02 20:50:27.018671 == Constructing image ==
2022-Jun-02 20:50:27.018687 Opening reordered part 0 spw 0 for /net/rijn2/data2/rvweeren/ELAIS_store/sub6asec_L686962_SB001_uv_12CFFDDA8t_140MHz.ms.sub.shift.avg.apply_infield.avg4ch.ms
2022-Jun-02 20:50:27.018718 Opening reordered part 0 spw 0 for /net/rijn2/data2/rvweeren/ELAIS_store/sub6asec_L686962_SB001_uv_12CFFDDA8t_140MHz.ms.sub.shift.avg.apply_infield.avg4ch.ms
2022-Jun-02 20:50:38.952566 Loading data in memory...
2022-Jun-02 20:50:39.262798 Loading data in memory...
2022-Jun-02 20:50:39.316337 Loading data in memory...
2022-Jun-02 20:50:39.368599 Loading data in memory...
wsclean -no-update-model-required -use-wgridder -minuv-l 80.0 -size 22500 22500 -weighting-rank-filter 3 -reorder -weight briggs -1.5 -parallel-reordering 6 -mgain 0.8 -data-column DATA -auto-mask 3 -auto-threshold 1.0 -pol i -name BDAimagetest_h5bavg -scale 0.4arcsec -taper-gaussian 1.2asec -niter 25000 -log-time -multiscale-scale-bias 0.8 -parallel-deconvolution 2048 -multiscale -multiscale-max-scales 8 -nmiter 1 -facet-regions facets.reg -parallel-gridding 4 -apply-facet-beam -facet-beam-update 600 -use-differential-lofar-beam -baseline-averaging 0.4 -apply-facet-solutions merged_withCS.h5 amplitude000,phase000/net/rijn2/data2/rvweeren/ELAIS_store/sub6asec_L686962_SB001_uv_12CFFDDA8t_140MHz.ms.sub.shift.avg.apply_infield.avg4ch.ms