CI tests with PTScotch not deterministic?
When I run make alltests with PTScotch installed (and DATAFILESPATH set) on my lap I get different output for
not ok diff-dm_impls_plex_tutorials-ex12_0_shell-True_ptscotch # Error code: 1 not ok diff-dm_impls_plex_tutorials-ex12_0_shell-False_ptscotch # Error code: 1
Looking in the PTScotch INSTALL.txt I see
When Scotch is compiled with the "-DCOMMON_RANDOM_FIXED_SEED" flag set, multiple executions of the same sequential programs will always yield the same results. Yet, non-deterministic multi-threaded and/or parallel algorithms may still introduce some variability in the produced results, due to operating system artifacts. To ease the tracking down of bugs, it is possible to coerce Scotch to exhibit a deterministic behavior by setting the "-DSCOTCH_DETERMINISTIC" flag. This option makes sure that two consecutive runs of Scotch on the same graph and same number of processes and/or threads will always yield the same result. However, it does not mean that two executions of Scotch on different numbers of processes and/or threads will always yield the same result. Indeed, changing the number of processes impacts graph data distribution and consequently the order in which vertices are processed. Consequently, coarsening algorithms are likely to produce different coarse graphs, leading to different partitions. This option is likely to have adverse impact on performance, since all non-deterministic multi-threaded algorithms will be disabled. In the case of Scotch, the multi-threaded matching algorithm for coarsening will be replaced by its sequential counterpart, whatever the available number of threads is. In the case of PT-Scotch, point-to-point messages will be received in a fixed order rather than being processed on a first-come, first-serve basis.
It is a pity it is not a PTScotch runtime option.
How should we fix this?