Use IMU for feature tracking initial estimates

Depends on !11 and basalt-headers!28

This makes feature tracking significantly more robust for sudden movements. See this video for a better understanding of the issue. See these before and after videos to see the improvement effects.

You can try it out with this dataset Calibration file odysseyplus_rt8_calib.json and, since the dataset was taken with a device with low overlap between its cameras, you'll need to use these on euroc_config.json:

"config.optical_flow_detection_nonoverlap": true,
"config.optical_flow_matching_guess_type": "REPROJ_AVG_DEPTH",

Results

The summary of the improvements are:

1. Increased feature count, especially on the right camera
2. Small ATE (absolute trajectory) improvement (w.r.t REPROJ_AVG_DEPTH)
3. Significant RTE (relative trajectory) improvements
4. Improved (lower) standard deviations in general
5. Worse times: probably due to more features being tracked, but be also aware that the timing info, while taken from the same computer (Ryzen 5800x), the two older compared versions were taken months ago. In any case it's still good enough to

Detailed metrics (from xrtslam-metrics)

• SAME_PIXEL: Before !7, numbers copied from that MR
• REPROJ_AVG_DEPTH: After !7, numbers copied from that MR
• OPTFLOW_IMU: After this MR

TUM-VI Rooms

Average (± stdev) pose estimation time [ms]

	OPTFLOW_IMU	REPROJ_AVG_DEPTH	SAME_PIXEL
TR1	4.77 ± 0.70	3.85 ± 0.61	3.80 ± 0.56
TR2	4.51 ± 0.61	3.74 ± 0.57	3.68 ± 0.54
TR3	4.67 ± 0.66	3.72 ± 0.60	3.62 ± 0.55
TR4	4.93 ± 0.79	3.98 ± 0.74	3.91 ± 0.66
TR5	4.72 ± 0.66	3.74 ± 0.59	3.69 ± 0.60
TR6	4.94 ± 0.69	4.10 ± 0.61	4.05 ± 0.57
[AVG]	4.76 ± 0.68	3.86 ± 0.62	3.79 ± 0.58

Average feature count for each camera

	OPTFLOW_IMU	REPROJ_AVG_DEPTH	SAME_PIXEL
TR1	[122 111] ± [35 32]	[100 85] ± [43 36]	[99 81] ± [42 34]
TR2	[113 94] ± [32 30]	[104 81] ± [34 29]	[102 77] ± [33 28]
TR3	[114 98] ± [37 35]	[97 77] ± [40 34]	[95 69] ± [39 30]
TR4	[129 114] ± [43 38]	[116 95] ± [47 37]	[113 88] ± [46 33]
TR5	[111 99] ± [33 29]	[84 71] ± [42 36]	[82 67] ± [41 33]
TR6	[144 125] ± [37 34]	[142 110] ± [38 31]	[140 105] ± [39 30]
[AVG]	[122 107] ± [36 33]	[107 86] ± [40 34]	[105 81] ± [40 31]

Average completion percentage [%]

	OPTFLOW_IMU	REPROJ_AVG_DEPTH	SAME_PIXEL
TR1	✓	✓	✓
TR2	✓	✓	✓
TR3	✓	✓	✓
TR4	✓	✓	✓
TR5	✓	✓	✓
TR6	✓	✓	✓
[AVG]	✓	✓	✓

Absolute trajectory error (ATE) [m]

	OPTFLOW_IMU	REPROJ_AVG_DEPTH	SAME_PIXEL
TR1	0.115 ± 0.051	0.112 ± 0.050	0.108 ± 0.049
TR2	0.075 ± 0.037	0.076 ± 0.039	0.078 ± 0.040
TR3	0.119 ± 0.055	0.118 ± 0.056	0.124 ± 0.057
TR4	0.053 ± 0.025	0.061 ± 0.031	0.058 ± 0.029
TR5	0.181 ± 0.072	0.175 ± 0.068	0.173 ± 0.067
TR6	0.018 ± 0.008	0.020 ± 0.008	0.021 ± 0.011
[AVG]	0.094 ± 0.041	0.094 ± 0.042	0.094 ± 0.042

Relative trajectory error (RTE) [m]

	OPTFLOW_IMU	REPROJ_AVG_DEPTH	SAME_PIXEL
TR1	0.007964 ± 0.004624	0.008697 ± 0.005210	0.008864 ± 0.005406
TR2	0.007560 ± 0.005115	0.007893 ± 0.005405	0.008017 ± 0.005465
TR3	0.006405 ± 0.004137	0.006758 ± 0.004357	0.006783 ± 0.004327
TR4	0.006046 ± 0.004037	0.006545 ± 0.004529	0.006608 ± 0.004590
TR5	0.009807 ± 0.005927	0.010911 ± 0.006807	0.010930 ± 0.006817
TR6	0.004055 ± 0.002290	0.003896 ± 0.002233	0.003914 ± 0.002234
[AVG]	0.006973 ± 0.004355	0.007450 ± 0.004757	0.007519 ± 0.004806

EuRoC

Average (± stdev) pose estimation time [ms]

	OPTFLOW_IMU	REPROJ_AVG_DEPTH	SAME_PIXEL
EMH01	9.08 ± 1.89	5.64 ± 3.42	5.91 ± 3.75
EMH02	9.17 ± 2.01	6.28 ± 4.25	11.72 ± 16.74
EMH03	8.58 ± 1.75	5.83 ± 2.43	6.50 ± 3.59
EMH04	8.05 ± 1.47	5.70 ± 2.21	5.81 ± 2.41
EMH05	8.12 ± 1.54	5.91 ± 2.62	5.69 ± 2.02
EV101	7.02 ± 1.31	5.66 ± 1.12	5.66 ± 1.13
EV102	7.27 ± 1.72	4.54 ± 1.25	4.63 ± 1.37
EV103	7.08 ± 1.45	3.99 ± 1.28	4.05 ± 1.37
EV201	7.44 ± 1.59	8.22 ± 8.24	5.29 ± 2.74
EV202	7.21 ± 1.47	4.41 ± 1.30	4.52 ± 1.44
[AVG]	7.90 ± 1.62	5.62 ± 2.81	5.98 ± 3.66

Average feature count for each camera

	OPTFLOW_IMU	REPROJ_AVG_DEPTH	SAME_PIXEL
EMH01	[155 115] ± [77 64]	[153 101] ± [79 59]	[157 107] ± [74 51]
EMH02	[162 116] ± [81 70]	[160 105] ± [83 60]	[161 108] ± [82 58]
EMH03	[146 124] ± [65 47]	[140 108] ± [67 43]	[140 105] ± [68 42]
EMH04	[129 115] ± [58 50]	[127 99] ± [59 45]	[126 94] ± [59 43]
EMH05	[134 117] ± [60 47]	[132 103] ± [62 44]	[131 97] ± [62 42]
EV101	[163 137] ± [50 45]	[160 124] ± [50 41]	[157 117] ± [50 38]
EV102	[107 95] ± [46 38]	[91 74] ± [46 35]	[90 72] ± [46 34]
EV103	[60 53] ± [41 37]	[51 41] ± [42 35]	[51 40] ± [42 34]
EV201	[131 95] ± [54 43]	[130 88] ± [54 42]	[130 87] ± [54 42]
EV202	[91 68] ± [47 34]	[83 58] ± [49 33]	[83 57] ± [49 32]
[AVG]	[128 104] ± [58 47]	[123 90] ± [59 43]	[122 88] ± [59 42]

Average completion percentage [%]

	OPTFLOW_IMU	REPROJ_AVG_DEPTH	SAME_PIXEL
EMH01	✓	✓	✓
EMH02	✓	✓	✓
EMH03	✓	✓	✓
EMH04	✓	✓	✓
EMH05	✓	✓	✓
EV101	✓	✓	✓
EV102	✓	✓	✓
EV103	✓	✓	✓
EV201	✓	✓	✓
EV202	✓	✓	✓
[AVG]	✓	✓	✓

Absolute trajectory error (ATE) [m]

	OPTFLOW_IMU	REPROJ_AVG_DEPTH	SAME_PIXEL
EMH01	0.075 ± 0.021	0.068 ± 0.020	0.066 ± 0.023
EMH02	0.049 ± 0.020	0.056 ± 0.032	0.054 ± 0.025
EMH03	0.065 ± 0.022	0.067 ± 0.021	0.062 ± 0.019
EMH04	0.119 ± 0.042	0.112 ± 0.039	0.114 ± 0.038
EMH05	0.154 ± 0.040	0.179 ± 0.056	0.145 ± 0.041
EV101	0.043 ± 0.017	0.043 ± 0.018	0.043 ± 0.017
EV102	0.053 ± 0.016	0.047 ± 0.014	0.045 ± 0.013
EV103	0.055 ± 0.014	0.051 ± 0.019	0.053 ± 0.020
EV201	0.038 ± 0.013	0.043 ± 0.017	0.039 ± 0.015
EV202	0.037 ± 0.013	0.046 ± 0.021	0.049 ± 0.021
[AVG]	0.069 ± 0.022	0.071 ± 0.026	0.067 ± 0.023

Relative trajectory error (RTE) [m]

	OPTFLOW_IMU	REPROJ_AVG_DEPTH	SAME_PIXEL
EMH01	0.005143 ± 0.002933	0.005195 ± 0.003052	0.005268 ± 0.003102
EMH02	0.004247 ± 0.002296	0.004215 ± 0.002234	0.004434 ± 0.002311
EMH03	0.011736 ± 0.007583	0.012011 ± 0.007753	0.011958 ± 0.007691
EMH04	0.012585 ± 0.007979	0.012614 ± 0.008074	0.012563 ± 0.008073
EMH05	0.010774 ± 0.006223	0.010670 ± 0.006175	0.010958 ± 0.006219
EV101	0.013034 ± 0.006259	0.013042 ± 0.006260	0.013037 ± 0.006251
EV102	0.011103 ± 0.004762	0.011650 ± 0.005077	0.011768 ± 0.005152
EV103	0.012403 ± 0.006149	0.013096 ± 0.006473	0.013341 ± 0.006866
EV201	0.003396 ± 0.001861	0.003481 ± 0.001983	0.003452 ± 0.001952
EV202	0.007195 ± 0.003950	0.009049 ± 0.005867	0.009121 ± 0.005971
[AVG]	0.009162 ± 0.005000	0.009502 ± 0.005295	0.009590 ± 0.005359