Need to update `trials.stop_time` to include Plane_3 `timestamps`
Issue
For example, the data/Npy/test.nwb
generated with calimag tests/user.config.toml
, for each trial, the last timestamp of Plane_3
exceeds.
from pynwb import NWBHDF5IO
data_path = 'data/Npy/test.nwb'
nwb = NWBHDF5IO(data_path, "r").read()
F_data = nwb.processing['ophys']['Fluorescence'].roi_response_series
dT_trial_min = 30
t_F = {p: F.timestamps[:] for p,F in F_data.items()}
reset_tF = {p: np.where(np.diff(t) > dT_trial_min)[0]+1
for p, t in t_F.items()}
For example (focus on 2nd column), with trial=0
the last timestamp before big jump for Plane_3
is 30.06073307
but nwb.trials.stop_time
is 29.999
. Similarly with trial=57
, 3450.11078353 > 3450.049
.
In [64]: trial = 0
In [65]: {p: t_F[p][reset_tF[p][trial]-2:reset_tF[p][trial]+3] for p in t_F}
Out[65]:
{'Plane_1': array([29.69143659, 29.91301448, 60.006 , 60.22757813, 60.44915625]),
'Plane_2': array([29.76529589, 29.98687377, 60.07985937, 60.3014375 , 60.52301562]),
'Plane_3': array([29.83915518, 30.06073307, 60.15371875, 60.37529687, 60.596875 ])}
^^^^ this
In [66]: nwb.trials[trial]
Out[66]:
id start_time stop_time
0 0.0 29.999
In [67]: trial = 57
In [68]: {p: t_F[p][reset_tF[p][trial]-2:reset_tF[p][trial]+3] for p in t_F}
Out[68]:
{'Plane_1': array([3449.74148644, 3449.96306469, 3480.054 , 3480.2755782 , 3480.49715641]),
'Plane_2': array([3449.81534586, 3450.03692411, 3480.1278594 , 3480.3494376 , 3480.57101581]),
'Plane_3': array([3449.88920528, 3450.11078353, 3480.2017188 , 3480.42329701, 3480.64487521])}
^^^^ this
In [69]: nwb.trials[trial]
Out[69]:
id start_time stop_time
57 3420.05 3450.049
Evaluation of issue
Right now - Priority LOW
This is not a big deal (right now) but (e.g.) for people who analyze the data by concatenating all 3 planes and yet base the per-plane timestamps to separate trials instead of the first plane's, there will be a mismatch in dimension and the error might not be so easily spotted.
Even with analyzing each plane alone, it would be confusing when comparing between the nwb
object stop_time
with the plane's own timestamps.
Q1 Is there some information about trial separation from suite2p/*.npy
files? If so, it's easier
Q2 Would there also similar issue when parsing Chan2
? I suspect not.
Later on
As of now, from https://gitlab.com/fleischmann-lab/calcium-imaging/calimag/-/blob/master/src/calimag/nwb_converter.py#L110-112, the trials are added based on the teensy data.
However, later on with more data sources with different sampling rates, and if some files do not contain the trial separation or indication, this might be a more serious issue.
Solution
Currently I'm not sure what's the most generalized solution is. For now, maybe the inter-trial time difference (like dT_trial_min
, can be deduced from other files processed before or supplied by user) can be used to support the separation to modify stop_time
at the end.