-
Rodrigo Tobar authored
The new Low ICD format is quite different from Mid's: it contains more items, and most of them are only sent during the start-of-stream heap, with data heaps only carrying the time and correlator output data. There are also items that don't exist anymore because they don't make sense in Low, plus a new way to encode time in each correlator dump. Overall, implementing this new ICD marks a clear break from the ICD symmetry we had until now between Low and Mid. This commit adds a new user-facing "transmission.telescope" option that controls the telescope's ICD to follow when generating data for a SPEAD stream. When using Mid, and since Mid's ICD hasn't changed, the existing emulation is used, with the exception that now we send CCI data as part of the correlator output data item (we hadn't done this properly until now, but never cared about it because the receiver doesn't use this data). When using Low the new behavior is triggered, with most of the new Items' values sent in the start-of-stream heaps. Since in receive-core both ICDs follow a common interface, some of the differences in the code at this level can be handled without the need of further if/else branches, which simplifies the code somewhat. A new unit test also checks that sending both Mid and Low data is successful in a small 1-data-heap run, and that the receiver gets the items it's supposed to get with each heap type. In order to be able to use the new ICD definitions we have upgraded our dependency on receive-ore to >= 4.0.0. This new version brings breaking changes, not only in how the ICDs and their items are defined, but also in that it has a clearer separation between the utility code dealing with Measurement Set I/O and the ICD, so we have adjusted our usage of those APIs as well. The new version of receive-core also requires Python >= 3.10 so we need to upgrade our own Python version dependency constraints. A particularity about the Low ICD is that it assumes a single channel per SPEAD stream, and therefore doesn't send any item indicating the number of channels. Our emulator follows suit, assuming 1 channel per stream by default when using the Low telescope, can this behavior can still be overridden by the user. If they do, then a warning is issued noting the non-standard behavior being triggered, but still no Item is sent indicating this; instead the receiver can only infer this information by inspecting the correlator data output shape to deduce the number of channels sent. Although this implementation already follows the new ICD, there are some details that have been left out of this first iteration, namely: * The heap counter doesn't encode any information, partly because the python bindings for spead2 don't allow to set the heap counter. * Some of the data sent on the start-of-stream heap is hardcoded to dummy values instead of using proper ones. This requires some further plumbing (new user options, read more data from the input Measurement Set), and since at the moment the data is simply logged on the receiver there is no rush to get it right yet. * There is no re-sending of start-of-stream heaps. This work is part of YAN-1273.