Skip to content
  • Rodrigo Tobar's avatar
    Implement emulation of the new Low ICD format · 28c61d20
    Rodrigo Tobar authored
    The new Low ICD format is quite different from Mid's: it contains more
    items, and most of them are only sent during the start-of-stream heap,
    with data heaps only carrying the time and correlator output data. There
    are also items that don't exist anymore because they don't make sense in
    Low, plus a new way to encode time in each correlator dump. Overall,
    implementing this new ICD marks a clear break from the ICD symmetry we
    had until now between Low and Mid.
    
    This commit adds a new user-facing "transmission.telescope" option that
    controls the telescope's ICD to follow when generating data for a SPEAD
    stream. When using Mid, and since Mid's ICD hasn't changed, the existing
    emulation is used, with the exception that now we send CCI data as part
    of the correlator output data item (we hadn't done this properly until
    now, but never cared about it because the receiver doesn't use this
    data). When using Low the new behavior is triggered, with most of the
    new Items' values sent in the start-of-stream heaps. Since in
    receive-core both ICDs follow a common interface, some of the
    differences in the code at this level can be handled without the need of
    further if/else branches, which simplifies the code somewhat. A new unit
    test also checks that sending both Mid and Low data is successful in a
    small 1-data-heap run, and that the receiver gets the items it's
    supposed to get with each heap type.
    
    In order to be able to use the new ICD definitions we have upgraded our
    dependency on receive-ore to >= 4.0.0. This new version brings breaking
    changes, not only in how the ICDs and their items are defined, but also
    in that it has a clearer separation between the utility code dealing
    with Measurement Set I/O and the ICD, so we have adjusted our usage of
    those APIs as well. The new version of receive-core also requires Python
    >= 3.10 so we need to upgrade our own Python version dependency
    constraints.
    
    A particularity about the Low ICD is that it assumes a single channel
    per SPEAD stream, and therefore doesn't send any item indicating the
    number of channels. Our emulator follows suit, assuming 1 channel per
    stream by default when using the Low telescope, can this behavior can
    still be overridden by the user. If they do, then a warning is issued
    noting the non-standard behavior being triggered, but still no Item is
    sent indicating this; instead the receiver can only infer this
    information by inspecting the correlator data output shape to deduce the
    number of channels sent.
    
    Although this implementation already follows the new ICD, there are some
    details that have been left out of this first iteration, namely:
    
     * The heap counter doesn't encode any information, partly because the
       python bindings for spead2 don't allow to set the heap counter.
     * Some of the data sent on the start-of-stream heap is hardcoded to
       dummy values instead of using proper ones. This requires some further
       plumbing (new user options, read more data from the input Measurement
       Set), and since at the moment the data is simply logged on the
       receiver there is no rush to get it right yet.
     * There is no re-sending of start-of-stream heaps.
    
    This work is part of YAN-1273.
    28c61d20