alternative data representation
In the current implementation data is represented by storing values and 'metadat' (such as confidence) colocated next to each other:
This makes it convinient as each datapoint can be handled as a c-struct. An alternative approach would be to colocate all datapoints and co-locate all metadata points:
|---data(1)--|--mdata(1)--|---data(2)--|--mdata(2)--| => |---data(1)--|---data(2)--|--mdata(1)--|--mdata(2)--|
This would create two 'arrays' to store our data (so a outer struct) instead of inner structs. The advantage here would be that with this kind of storage we will be able to use vector math on the data and metadata vector potentially improving performance significantly - gcc might itself be smart enough to optimize normal looks this way.
If we also store the 'lenght' of the data in the outer struct (sacrifice a few extra bit) we could also very efficiently use this kind of structure when allowing prefixed opperations as we would be able to perform a number of functions that only reduce the points in place only allocating/copying the data at the last step when it's serialized again.