Secondary transform doesn't update buffers correctly leading to suboptimal mode decisions in the encoder.
What version / commit were you testing with?
research-v4.0.0-rc1-20-g ae397e62
What steps will reproduce the problem?
- Please read the code.
What is the expected output? Correct optimization of the secondary transform mode.
What do you see instead? Occasional suboptimal secondary transform mode decisions.
Please use labels and text to provide additional information.
Background: For intra blocks encoder decides transform sizes/types using search_tx_type(). search_tx_type() in turn will occasionally evaluate distortion in pixel domain using dist_block_px_domain(). dist_block_px_domain calls av1_inverse_transform_block() which inverts the secondary and then the primary transform.
Issue: During av1_inverse_transform_block() operation the secondary transforms inversion updates the transform coefficients. Hence after the call to dist_block_px_domain() within search_tx_type() the transform coefficients are in updated state.
Manifestation: This issue creates a problem at the end of search_tx_type() where the wrong (updated) coefficients are used in recon_intra() in order to enable intra prediction for the remaining transform blocks. This likely leads to suboptimal secondary transform decisions occasionally (when search_tx_type() chooses to use dist_block_px_domain()).
Solution: Please buffer the relevant coefficients as needed so that coefficients are not inappropriately updated.