Correlated (e.g. NLO) event handling and bin smearing
As reported on the Rivet mailing list, the handling of NLO events is not functional in Rivet 3.x yet. In particular, two features are entangled as described in App. A of the Rivet3 paper and that might be part of the problem.
Let me try to disentangle them and give my point of view here, such that others can chime in:
- Statistical treatment of correlated events (event groups defined by the same event number)
- This can be used for NLO subtraction events, but not only those, so should be a more general feature.
- The main point is that fills into the same bin stemming from events within the same group should be combined and filled once. This will produce the correct statistics for those 100% correlated events.
- I had provided a "plugin" for Rivet 1.x and 2.x as a workaround to make this possible in custom analyses (https://gitlab.com/hepcedar/rivetcontrib/-/tree/master/nlohisto) but we postponed a proper implementation to Rivet 3.x.
- Bin smearing to avoid misbinning effects
- This is specific to fixed-order N(N)LO, where subtraction events can appear, which have different kinematics than their real event. If they correspond to a collinear/soft configuration, their weights will be +-∞ and this won't cancel anymore if they end up in different bins.
- For IR-safe observables, these misbinnings will never happen for exactly collinear/soft configurations, so there won't be +-∞ in the histograms. But they can happen when approaching the soft/collinear limits while close to a bin boundary, and induce large weights in different bins.
- One can think about smart filling procedures like:
- the fractional filling, which effectively smears such events across bins, as described in App. A of Rivet3, or
- moving fractional weights between the correlated events making use of the knowledge of their divergence structure (see e.g. https://sherpa.hepforge.org/doc/SHERPA-MC-2.2.8.html#Avoiding-misbinning-effects)
The 1st point is more urgent to address, since we are really getting the statistics wrong at the moment.
The 2nd point can be argued to not really be in the scope (or even capabilities?) of an analysis tool to solve. In particular, it modifies the prediction and doesn't give a faithful analysis of the provided events. So if and when we fix it, it should at least be configurable on an opt-in basis, and definitely not be applied automatically to event groups in my opinion. There are several pitfalls for this (e.g. overflow/underflow).
Is it possible to disable (2) for now and fix (1)? Tagging @lonnblad for his opinion, since he has worked on this.