CI tests with PTScotch not deterministic?
When I run make alltests with PTScotch installed (and DATAFILESPATH set) on my lap I get different output for
not ok diff-dm_impls_plex_tutorials-ex12_0_shell-True_ptscotch # Error code: 1
not ok diff-dm_impls_plex_tutorials-ex12_0_shell-False_ptscotch # Error code: 1
Looking in the PTScotch INSTALL.txt I see
When Scotch is compiled with the "-DCOMMON_RANDOM_FIXED_SEED"
flag set, multiple executions of the same sequential programs
will always yield the same results. Yet, non-deterministic
multi-threaded and/or parallel algorithms may still introduce
some variability in the produced results, due to operating system
artifacts.
To ease the tracking down of bugs, it is possible to coerce Scotch to
exhibit a deterministic behavior by setting the
"-DSCOTCH_DETERMINISTIC" flag. This option makes sure that two
consecutive runs of Scotch on the same graph and same number of
processes and/or threads will always yield the same result. However,
it does not mean that two executions of Scotch on different numbers of
processes and/or threads will always yield the same result. Indeed,
changing the number of processes impacts graph data distribution and
consequently the order in which vertices are processed. Consequently,
coarsening algorithms are likely to produce different coarse graphs,
leading to different partitions.
This option is likely to have adverse impact on performance, since all
non-deterministic multi-threaded algorithms will be disabled. In the
case of Scotch, the multi-threaded matching algorithm for coarsening
will be replaced by its sequential counterpart, whatever the available
number of threads is. In the case of PT-Scotch, point-to-point
messages will be received in a fixed order rather than being processed
on a first-come, first-serve basis.
It is a pity it is not a PTScotch runtime option.
How should we fix this?