H.264 parsing incorrect
Hello,
looking at your solution for #1704 (closed) I realized that you seem to think that SPS and PPS indicate the start of a new access unit (or at least that they last VCL NALU before them was the last VCL NALU of the preceding access unit). They don't. In fact, section 7.4.1.2.3 contains that SPS and PPS signifies that the old access unit has ended, but only if it comes after the last VCL NALU of the preceding primary coded picture. So one is allowed to insert SPS and PPS between the slices of a primary coded picture. Note 2 of this section even says this: "Sequence parameter set NAL units or picture parameter set NAL units may be present in an access unit, but cannot follow the last VCL NAL unit of the primary coded picture within the access unit, as this condition would specify the start of a new access unit." Of course one is not allowed to do arbitrary things with SPS and PPS in the middle of a picture: One mustn't change (overwrite) the active SPS/PPS, but one can repeat the currently active SPS and PPS and one may send SPS and PPS with different id's than the currently active SPS and PPS have. (These restrictions are part of 7.4.1.2.1.)
In order to find out what the first VCL NALU of a primary coded picture is (which also shows what the last VCL NALU of the preceding access unit was) one can use the criteria of 7.4.1.2.4; some of the non-VCL-NALUs can aid in determining the boundaries of access units, namely AUD and SEI (they shall precede the primary coded picture and therefore indicate that the last VCL NALU was part of the last access unit).
I have uploaded a typical Bluray sample that uses multiple slices and added the SPS/PPS in between the slices. "Multiple.Slices.mkv" is the original file, "Multiple.Slices.2.mkv" is the file with SPS and PPS inserted between slices of the first (IDR) frame; multiple.slices.2.264 is extracted from "Multiple.Slices.2.mkv" using mkvextract (latest prebuild 1610). Upon remuxing this file mkvmerge splits the first IDR frame and puts it into two blocks. ffmpeg complains upon decoding; it, too, seems to interpret SPS and PPS as starting a new access unit -- it even complains about multiple.slices.2.264. Notice that h264_parse seems to follow the standard and the recipe in 7.4.1.2.4 literally. See the "Nal is part of last picture" and "Nal is new picture" messages (I have trimmed the scaling lists in the PPS a bit):
Nal length 56 start code 4 bytes
ref 3 type 7 Sequence parameter set
profile: 100
constaint_set0_flag: 0
constaint_set1_flag: 0
constaint_set2_flag: 0
constaint_set3_flag: 0
level_idc: 41
seq parameter set id: 0
chroma format idx: 1
bit depth luma minus8: 0
bit depth chroma minus8: 0
Qpprime Y Zero Transform Bypass flag: 0
Seq Scaling Matrix Present Flag: 0
log2_max_frame_num_minus4: 12
pic_order_cnt_type: 0
log2_max_pic_order_cnt_lsb_minus4: 12
num_ref_frames: 4
gaps_in_frame_num_value_allowed_flag: 0
pic_width_in_mbs_minus1: 119 (1920)
pic_height_in_map_minus1: 67
frame_mbs_only_flag: 1
derived height: 1088
direct_8x8_inference_flag: 1
frame_cropping_flag: 1
frame_crop_left_offset: 0
frame_crop_right_offset: 0
frame_crop_top_offset: 0
frame_crop_bottom_offset: 4
vui_parameters_present_flag: 1
aspect_ratio_info_present_flag: 1
aspect_ratio_idc:1
overscan_info_present_flag: 0
video_signal_info_present_flag: 0
chroma_loc_info_present_flag: 0
timing_info_present_flag: 1
num_units_in_tick: 1001
time_scale: 48000
fixed_frame_scale: 1
nal_hrd_parameters_present_flag: 1
cpb_cnt_minus1: 0
bit_rate_scale: 1
cpb_size_scale: 3
bit_rate_value_minus1[0]: 289061
cpb_size_value_minus1[0]: 234374
cbr_flag[0]: 0
initial_cpb_removal_delay_length_minus1: 17
cpb_removal_delay_length_minus1: 17
dpb_output_delay_length_minus1: 5
time_offset_length: 24
vcl_hrd_parameters_present_flag: 1
cpb_cnt_minus1: 0
bit_rate_scale: 1
cpb_size_scale: 3
bit_rate_value_minus1[0]: 289061
cpb_size_value_minus1[0]: 234374
cbr_flag[0]: 0
initial_cpb_removal_delay_length_minus1: 17
cpb_removal_delay_length_minus1: 17
dpb_output_delay_length_minus1: 5
time_offset_length: 24
low_delay_hrd_flag: 0
pic_struct_present_flag: 1
motion_vectors_over_pic_boundaries_flag: 1
max_bytes_per_pic_denom: 0
max_bits_per_mb_denom: 0
log2_max_mv_length_horizontal: 13
log2_max_mv_length_vertical: 11
num_reorder_frames: 4
max_dec_frame_buffering: 4
Nal length 69 start code 4 bytes
ref 3 type 8 Picture parameter set
pic_parameter_set_id: 0
seq_parameter_set_id: 0
entropy_coding_mode_flag: 1
pic_order_present_flag: 0
num_slice_groups_minus1: 0
num_ref_idx_l0_active_minus1: 1
num_ref_idx_l1_active_minus1: 0
weighted_pred_flag: 0
weighted_bipred_idc: 0
pic_init_qp_minus26: 0
pic_init_qs_minus26: 0
chroma_qp_index_offset: -1
deblocking_filter_control_present_flag: 1
constrained_intra_pred_flag: 0
redundant_pic_cnt_present_flag: 0
transform_8x8_mode_flag: 1
pic_scaling_matrix_present_flag: 1
Pic Scaling List[0] Present Flag: 1
...
scaling list[7][63]: 19
second_chroma_qp_index_offset: 4294967295
Nal length 18 start code 4 bytes
ref 0 type 6 SEI
payload_type: 0 buffering_period
payload_size: 10
0xa1 0xd9 0x80 0x0 0x2 0x1d 0x98 0x0
0x0 0x40
seq_parameter_set_id: 0
initial_cpb_removal_delay[0]: 69324
initial_cpb_removal_delay_offset[0]: 0
initial_cpb_removal_delay[0]: 69324
initial_cpb_removal_delay_offset[0]: 0
Nal length 20 start code 4 bytes
ref 0 type 6 SEI
payload_type: 1 pic_timing
payload_size: 12
0x0 0x0 0x2 0x9 0x8 0x0 0xf7 0xee
0x0 0x0 0x0 0x8
cpb_removal_delay: 0
dpb_output_delay: 2
pict_struct: 0
clock_timestamp_flag[0]: 1
ct_type: 0
nuit_field_base_flag: 1
counting_type: 1
full_timestamp_flag: 0
discontinuity_flag: 0
cnt_dropped_flag: 0
n_frame: 0
seconds_flag: 1
seconds_value: 59
minutes_flag: 1
minutes_value: 59
hours_flag: 1
hours_value: 0
time_offset: 0
Nal length 9 start code 4 bytes
ref 0 type 6 SEI
payload_type: 6 recovery_point
payload_size: 1 0xc4
recovery_frame_cnt: 0
exact_match_flag: 1
broken_link_flag: 0
changing_slice_group_idc: 0
Nal length 923 start code 4 bytes
ref 3 type 5 Coded slice of an IDR picture
first_mb_in_slice: 0
slice_type: 7 (I)
pic_parameter_set_id: 0
frame_num: 0 (16 bits)
idr_pic_id: 0
pic_order_cnt_lsb: 0
Nal length 56 start code 4 bytes
ref 3 type 7 Sequence parameter set
profile: 100
constaint_set0_flag: 0
constaint_set1_flag: 0
constaint_set2_flag: 0
constaint_set3_flag: 0
level_idc: 41
seq parameter set id: 0
chroma format idx: 1
bit depth luma minus8: 0
bit depth chroma minus8: 0
Qpprime Y Zero Transform Bypass flag: 0
Seq Scaling Matrix Present Flag: 0
log2_max_frame_num_minus4: 12
pic_order_cnt_type: 0
log2_max_pic_order_cnt_lsb_minus4: 12
num_ref_frames: 4
gaps_in_frame_num_value_allowed_flag: 0
pic_width_in_mbs_minus1: 119 (1920)
pic_height_in_map_minus1: 67
frame_mbs_only_flag: 1
derived height: 1088
direct_8x8_inference_flag: 1
frame_cropping_flag: 1
frame_crop_left_offset: 0
frame_crop_right_offset: 0
frame_crop_top_offset: 0
frame_crop_bottom_offset: 4
vui_parameters_present_flag: 1
aspect_ratio_info_present_flag: 1
aspect_ratio_idc:1
overscan_info_present_flag: 0
video_signal_info_present_flag: 0
chroma_loc_info_present_flag: 0
timing_info_present_flag: 1
num_units_in_tick: 1001
time_scale: 48000
fixed_frame_scale: 1
nal_hrd_parameters_present_flag: 1
cpb_cnt_minus1: 0
bit_rate_scale: 1
cpb_size_scale: 3
bit_rate_value_minus1[0]: 289061
cpb_size_value_minus1[0]: 234374
cbr_flag[0]: 0
initial_cpb_removal_delay_length_minus1: 17
cpb_removal_delay_length_minus1: 17
dpb_output_delay_length_minus1: 5
time_offset_length: 24
vcl_hrd_parameters_present_flag: 1
cpb_cnt_minus1: 0
bit_rate_scale: 1
cpb_size_scale: 3
bit_rate_value_minus1[0]: 289061
cpb_size_value_minus1[0]: 234374
cbr_flag[0]: 0
initial_cpb_removal_delay_length_minus1: 17
cpb_removal_delay_length_minus1: 17
dpb_output_delay_length_minus1: 5
time_offset_length: 24
low_delay_hrd_flag: 0
pic_struct_present_flag: 1
motion_vectors_over_pic_boundaries_flag: 1
max_bytes_per_pic_denom: 0
max_bits_per_mb_denom: 0
log2_max_mv_length_horizontal: 13
log2_max_mv_length_vertical: 11
num_reorder_frames: 4
max_dec_frame_buffering: 4
Nal length 69 start code 4 bytes
ref 3 type 8 Picture parameter set
pic_parameter_set_id: 0
seq_parameter_set_id: 0
entropy_coding_mode_flag: 1
pic_order_present_flag: 0
num_slice_groups_minus1: 0
num_ref_idx_l0_active_minus1: 1
num_ref_idx_l1_active_minus1: 0
weighted_pred_flag: 0
weighted_bipred_idc: 0
pic_init_qp_minus26: 0
pic_init_qs_minus26: 0
chroma_qp_index_offset: -1
deblocking_filter_control_present_flag: 1
constrained_intra_pred_flag: 0
redundant_pic_cnt_present_flag: 0
transform_8x8_mode_flag: 1
pic_scaling_matrix_present_flag: 1
Pic Scaling List[0] Present Flag: 1
...
scaling list[7][63]: 19
second_chroma_qp_index_offset: 4294967295
Nal length 1933 start code 4 bytes
ref 3 type 5 Coded slice of an IDR picture
first_mb_in_slice: 2040
slice_type: 7 (I)
pic_parameter_set_id: 0
frame_num: 0 (16 bits)
idr_pic_id: 0
pic_order_cnt_lsb: 0
Nal is part of last picture
Nal length 1648 start code 4 bytes
ref 3 type 5 Coded slice of an IDR picture
first_mb_in_slice: 4080
slice_type: 7 (I)
pic_parameter_set_id: 0
frame_num: 0 (16 bits)
idr_pic_id: 0
pic_order_cnt_lsb: 0
Nal is part of last picture
Nal length 1092 start code 4 bytes
ref 3 type 5 Coded slice of an IDR picture
first_mb_in_slice: 6120
slice_type: 7 (I)
pic_parameter_set_id: 0
frame_num: 0 (16 bits)
idr_pic_id: 0
pic_order_cnt_lsb: 0
Nal is part of last picture
Nal length 20 start code 4 bytes
ref 0 type 6 SEI
payload_type: 1 pic_timing
payload_size: 12
0x0 0x0 0x84 0x9 0x8 0x2 0xf7 0xee
0x0 0x0 0x0 0x8
cpb_removal_delay: 2
dpb_output_delay: 4
pict_struct: 0
clock_timestamp_flag[0]: 1
ct_type: 0
nuit_field_base_flag: 1
counting_type: 1
full_timestamp_flag: 0
discontinuity_flag: 0
cnt_dropped_flag: 0
n_frame: 2
seconds_flag: 1
seconds_value: 59
minutes_flag: 1
minutes_value: 59
hours_flag: 1
hours_value: 0
time_offset: 0
Nal length 285 start code 4 bytes
ref 2 type 1 Coded slice of non-IDR picture
first_mb_in_slice: 0
slice_type: 5 (P)
pic_parameter_set_id: 0
frame_num: 1 (16 bits)
pic_order_cnt_lsb: 2
Nal is new picture
Grüße Andi