Some IDR frames not seekable
Created by: mkver
Hello,
warning: Given your rant about H264 you should not read this report when you already are in a bad mood. I have found a stream in the wild (i.e. contrary to my previous issue i did not need to create the streams myself using highly unusual settings; this comes from a blu-ray of this movie. I cut and demuxed (using tsmuxer -- mkvmerge deletes the access unit delimiters (and maybe other things?) and I did not want this to influence my analysis of the stream; the issue is the same if I directly mux from the BD-playlist without tsmuxer) the relevant section of the stream into 11.1.264 (the original file (on the BD) is called 00011.m2ts) and used h264_parse on it. It's output is 11.1.txt. Upon inspection the stream looks different than the ones I encountered so far; but it also seems to be compliant with the specs. The problem lies in the Picture parameter sets (PPS) that are in the stream: They change during the stream. Here is a representative PPS:
Nal length 133 start code 4 bytes
ref 1 type 8 Picture parameter set
pic_parameter_set_id: 0
seq_parameter_set_id: 0
entropy_coding_mode_flag: 1
pic_order_present_flag: 0
num_slice_groups_minus1: 0
num_ref_idx_l0_active_minus1: 1
num_ref_idx_l1_active_minus1: 0
weighted_pred_flag: 0
weighted_bipred_idc: 0
pic_init_qp_minus26: -9
pic_init_qs_minus26: 0
chroma_qp_index_offset: 0
deblocking_filter_control_present_flag: 1
constrained_intra_pred_flag: 0
redundant_pic_cnt_present_flag: 0
transform_8x8_mode_flag: 1
pic_scaling_matrix_present_flag: 1
Pic Scaling List[0] Present Flag: 1
delta: -2
scaling list[0][0]: 6
delta: 1
scaling list[0][1]: 7
delta: 0
scaling list[0][2]: 7
delta: 3
scaling list[0][3]: 10
delta: -3
scaling list[0][4]: 7
delta: 3
scaling list[0][5]: 10
delta: 6
scaling list[0][6]: 16
delta: -5
scaling list[0][7]: 11
delta: 0
scaling list[0][8]: 11
delta: 1
scaling list[0][9]: 12
delta: 1
scaling list[0][10]: 13
delta: -1
scaling list[0][11]: 12
delta: 5
scaling list[0][12]: 17
delta: 3
scaling list[0][13]: 20
delta: 0
scaling list[0][14]: 20
delta: -4
scaling list[0][15]: 16
Pic Scaling List[1] Present Flag: 0
Pic Scaling List[2] Present Flag: 0
Pic Scaling List[3] Present Flag: 1
delta: 2
scaling list[3][0]: 10
delta: 3
scaling list[3][1]: 13
delta: 0
scaling list[3][2]: 13
delta: 15
scaling list[3][3]: 28
delta: -14
scaling list[3][4]: 14
delta: 14
scaling list[3][5]: 28
delta: 13
scaling list[3][6]: 41
delta: -9
scaling list[3][7]: 32
delta: 0
scaling list[3][8]: 32
delta: 9
scaling list[3][9]: 41
delta: 5
scaling list[3][10]: 46
delta: -5
scaling list[3][11]: 41
delta: 43
scaling list[3][12]: 84
delta: 27
scaling list[3][13]: 111
delta: 0
scaling list[3][14]: 111
delta: -95
scaling list[3][15]: 16
Pic Scaling List[4] Present Flag: 0
Pic Scaling List[5] Present Flag: 0
Pic Scaling List[6] Present Flag: 1
delta: 1
scaling list[6][0]: 9
delta: 0
scaling list[6][1]: 9
delta: 0
scaling list[6][2]: 9
delta: 1
scaling list[6][3]: 10
delta: -1
scaling list[6][4]: 9
delta: 1
scaling list[6][5]: 10
delta: 0
scaling list[6][6]: 10
delta: 0
scaling list[6][7]: 10
delta: 0
scaling list[6][8]: 10
delta: 0
scaling list[6][9]: 10
delta: 1
scaling list[6][10]: 11
delta: -1
scaling list[6][11]: 10
delta: 0
scaling list[6][12]: 10
delta: 0
scaling list[6][13]: 10
delta: 1
scaling list[6][14]: 11
delta: 5
scaling list[6][15]: 16
delta: -5
scaling list[6][16]: 11
delta: -1
scaling list[6][17]: 10
delta: 0
scaling list[6][18]: 10
delta: 1
scaling list[6][19]: 11
delta: 1
scaling list[6][20]: 12
delta: 3
scaling list[6][21]: 15
delta: -3
scaling list[6][22]: 12
delta: -1
scaling list[6][23]: 11
delta: 0
scaling list[6][24]: 11
delta: 0
scaling list[6][25]: 11
delta: 5
scaling list[6][26]: 16
delta: 5
scaling list[6][27]: 21
delta: 8
scaling list[6][28]: 29
delta: -8
scaling list[6][29]: 21
delta: -5
scaling list[6][30]: 16
delta: -5
scaling list[6][31]: 11
delta: 0
scaling list[6][32]: 11
delta: 2
scaling list[6][33]: 13
delta: 2
scaling list[6][34]: 15
delta: 5
scaling list[6][35]: 20
delta: 0
scaling list[6][36]: 20
delta: -4
scaling list[6][37]: 16
delta: -3
scaling list[6][38]: 13
delta: -2
scaling list[6][39]: 11
delta: 6
scaling list[6][40]: 17
delta: 5
scaling list[6][41]: 22
delta: 7
scaling list[6][42]: 29
delta: 2
scaling list[6][43]: 31
delta: -8
scaling list[6][44]: 23
delta: -4
scaling list[6][45]: 19
delta: -5
scaling list[6][46]: 14
delta: 2
scaling list[6][47]: 16
delta: 5
scaling list[6][48]: 21
delta: 1
scaling list[6][49]: 22
delta: -1
scaling list[6][50]: 21
delta: -1
scaling list[6][51]: 20
delta: 5
scaling list[6][52]: 25
delta: 8
scaling list[6][53]: 33
delta: 3
scaling list[6][54]: 36
delta: -9
scaling list[6][55]: 27
delta: 0
scaling list[6][56]: 27
delta: 9
scaling list[6][57]: 36
delta: 4
scaling list[6][58]: 40
delta: -9
scaling list[6][59]: 31
delta: 9
scaling list[6][60]: 40
delta: 5
scaling list[6][61]: 45
delta: 0
scaling list[6][62]: 45
delta: -29
scaling list[6][63]: 16
Pic Scaling List[7] Present Flag: 1
delta: 4
scaling list[7][0]: 12
delta: 1
scaling list[7][1]: 13
delta: 0
scaling list[7][2]: 13
delta: 2
scaling list[7][3]: 15
delta: -2
scaling list[7][4]: 13
delta: 2
scaling list[7][5]: 15
delta: 3
scaling list[7][6]: 18
delta: -2
scaling list[7][7]: 16
delta: 0
scaling list[7][8]: 16
delta: 2
scaling list[7][9]: 18
delta: 2
scaling list[7][10]: 20
delta: -2
scaling list[7][11]: 18
delta: -1
scaling list[7][12]: 17
delta: 1
scaling list[7][13]: 18
delta: 2
scaling list[7][14]: 20
delta: 50
scaling list[7][15]: 70
delta: -50
scaling list[7][16]: 20
delta: -1
scaling list[7][17]: 19
delta: 0
scaling list[7][18]: 19
delta: 1
scaling list[7][19]: 20
delta: 13
scaling list[7][20]: 33
delta: 31
scaling list[7][21]: 64
delta: -30
scaling list[7][22]: 34
delta: -13
scaling list[7][23]: 21
delta: 0
scaling list[7][24]: 21
delta: 0
scaling list[7][25]: 21
delta: 51
scaling list[7][26]: 72
delta: 91
scaling list[7][27]: 163
delta: 92
scaling list[7][28]: 255
delta: -85
scaling list[7][29]: 170
delta: -89
scaling list[7][30]: 81
delta: -58
scaling list[7][31]: 23
delta: 0
scaling list[7][32]: 23
delta: 13
scaling list[7][33]: 36
delta: 30
scaling list[7][34]: 66
delta: 85
scaling list[7][35]: 151
delta: 4
scaling list[7][36]: 155
delta: -84
scaling list[7][37]: 71
delta: -32
scaling list[7][38]: 39
delta: -14
scaling list[7][39]: 25
delta: 71
scaling list[7][40]: 96
delta: 94
scaling list[7][41]: 190
delta: 65
scaling list[7][42]: 255
delta: 0
scaling list[7][43]: 255
delta: -27
scaling list[7][44]: 228
delta: -108
scaling list[7][45]: 120
delta: -75
scaling list[7][46]: 45
delta: 35
scaling list[7][47]: 80
delta: 89
scaling list[7][48]: 169
delta: 23
scaling list[7][49]: 192
delta: -28
scaling list[7][50]: 164
delta: 124
scaling list[7][51]: 32
delta: -33
scaling list[7][52]: 255
delta: 0
scaling list[7][53]: 255
delta: 0
scaling list[7][54]: 255
delta: 0
scaling list[7][55]: 255
delta: 0
scaling list[7][56]: 255
delta: 0
scaling list[7][57]: 255
delta: 0
scaling list[7][58]: 255
delta: 0
scaling list[7][59]: 255
delta: 0
scaling list[7][60]: 255
delta: 0
scaling list[7][61]: 255
delta: 0
scaling list[7][62]: 255
delta: 11
scaling list[7][63]: 10
second_chroma_qp_index_offset: 0
The relevant parameter here is pic_init_qp_minus26
: This is the parameter that changes during the stream; the other PPS also have pic_parameter_set_id
0, i.e. the old PPS is overwritten according to section 7.4.1.2.1 of the H264 spec. This is done "in-band": Because all SPS and PPS have id 0, it is impossible to put the other PPS as private data in the header as Haali's Codec list implies.
Here is some info about the stream: It contains both IDR-frames and I-frames that are not IDR frames. It contains recovery points (45 of them) and 45 SPS and 47 PPS. There are two lone PPS within a GOP (actually both are in the same GOP -- search for a NAL with length 25243) and the other 45 PPS follow the 45 SPS and are then followed by recovery_point SEI messages (with pic_timing and buffering_period SEI messages inbetween); all of these have the exact_match_flag
equal to 1 and are therefore treated as keyframes by mkvmerge. I have not found I frames that are not recovery_points.
The actual problem lies in the IDR frames which are not recovery_points and are not accompanied by PPS. They are treated as keyframes by mkvmerge although the pic_init_qp_minus26
-parameter which is needed for decoding is missing. So these are actually no points where one can seek to. And indeed one can't. Here are some images of what happens when one seeks to the wrong place (the grey images come from software decoding using VLC and MPC-HC with the internal LAV-filters (that both show grey is probably a result of both of them relying on libav/ffmpeg?); the green pictures are VLC with DXVA with Intel hardware decoding; the other two are MPC-HC with Intel hardware decoding (it doesn't matter whether DXVA (native), DXVA (copyback) or Intel Quick Sync)). I also uploaded them to your ftp-server.
Watching it without seeking works fine because in this way the player gets the right PPS (in particular, it works on a blu-ray; seeking to the chapter points works fine: They seem to have inserted SPS and PPS at these places). If I manually (i.e. via a hex editor) add the right PPS to the IDR-frame at 5.589 and then merge the resulting file 11.2.264 to 11.2.mkv, seeking to this keyframe works fine. I haven't fixed the other IDR-frames without PPS.
There are two possible ways to fix this in matroska: a) Either add the relevant SPS/PPS to the keyframe or b) make the frame non-seekable (i.e. don't set the keyframe-flag). The first option would be preferable of course (some players will, when told to seek to an IDR-frame without keyframe-flag in matroska see that this is an IDR frame and try to decode it directly and fail to do so; by the way, having more seek points is always preferable). a) will be quite a nontrivial improvement, I suppose. But contrary to some other things (like putting the audio that should be played
Some further thoughts:
- Every seekable frame has a recovery_point SEI message and exact_match_flag. Is it possible that the BD-spec actually say that only frames with recovery_point and exact_match_flag are seekable and not IDR-frames in general? And do seekable Blu-ray frames have to have these flags? If this is so, then there would be a workaround for this problem: Make a Blu-ray mode in which only such frames will be made matroska keyframes. One could make this mode user selectable or just use it as default for all m2ts and mpls files.
- This seems to happen only when there are a lot of keyframes in a short time (i.e. if there is a lot of motion so that one adds keyframes). Do the Blu-ray specs say that there can be not more than two seek points in every second?
- Given that this is a Sony release, we can safely assume that the encode was done with an old version of Sony's tools. Hopefully the guys doing BD-authoring use x264 now. (Even when the answer to 2. is yes, x264 would hopefully don't create encodes where the PPS change; at least I have never seen an x264 encode in which SPS or PPS change midstream.)
- Does matroska officially support overwriting SPS and PPS (i.e. different SPS and PPS with the same id)? These are necessarily transmitted in-band, as opposed to the out-of-band transmission in the codec private field in the header. Does mkvmerge handle SPS and PPS with different ids correctly (i.e. puts them in the codec private field)? I don't think so, because mkvmerge does not join h264-tracks with different codec private field.
- This stream has two consecutive IDR frames; luckily there are some SEI NAL-units inbetween and the idr_pic_id does not switch between 0 and 1 (as with newer versions of x264) but simply increases from 346 to 347. But it shows that #1704 might not be so artificial after all.
Grüße Andi
PS: I screwed it up a little during the upload process: Both the /bug and /Bug-folder belong to this issue. Windows-customs, you know.