Mismatch issue in quantize unit test
Created by: Cidana-Developers
Step to reproduce:
./SvtAv1UnitTests --gtest_filter="*Quantize*"
Console output:
[ RUN ] Quant/QuantizeBTest.input_random_all_q_all/0
/Users/yons/work/SVT-AV1/test/QuantAsmTest.cc:200: Failure
Expected equality of these values:
qcoeff_ref_[j]
Which is: 2978
qcoeff_test_[j]
Which is: 2979
Q mismatch at position: 109, Q: 4 coeff: 32766
/Users/yons/work/SVT-AV1/test/QuantAsmTest.cc:200: Failure
Expected equality of these values:
qcoeff_ref_[j]
Which is: 1424
qcoeff_test_[j]
Which is: 1425
Q mismatch at position: 24, Q: 16 coeff: 32767
/Users/yons/work/SVT-AV1/test/QuantAsmTest.cc:200: Failure
Expected equality of these values:
qcoeff_ref_[j]
Which is: -1424
qcoeff_test_[j]
Which is: -1425
Q mismatch at position: 59, Q: 16 coeff: -32767
/Users/yons/work/SVT-AV1/test/QuantAsmTest.cc:200: Failure
Expected equality of these values:
qcoeff_ref_[j]
Which is: 1023
qcoeff_test_[j]
Which is: 1024
Q mismatch at position: 128, Q: 25 coeff: 32761
/Users/yons/work/SVT-AV1/test/QuantAsmTest.cc:200: Failure
Expected equality of these values:
qcoeff_ref_[j]
Which is: 574
qcoeff_test_[j]
Which is: 575
Q mismatch at position: 65, Q: 50 coeff: 32765
/Users/yons/work/SVT-AV1/test/QuantAsmTest.cc:200: Failure
Target functions:
eb_aom_highbd_quantize_b_64x64_avx2
eb_aom_highbd_quantize_b_32x32_avx2
eb_aom_highbd_quantize_b_avx2
It happens when the coefficient are big and the difference is always 1.