Introduce --max-tx-size

Description

--max-tx-size allows the encoder to restrict selection of blocks transform sizes up to a maximum size. Valid values are 32 and 64.

In the AV1 standard, 64-pt transforms have the last 32 highest-frequency coefficients zeroed out during encoding, which means coded blocks can look visually blurry, especially when encoding fine noise-like textures.

PSNR and SSIM-based RDO metrics don't seem to detect this blurriness, so this setting combats this issue by not allowing 64-pt transforms to be considered in the first place. The result is an overall increase in output quality consistency, especially for still images in the medium to high quality range.

Example:

--max-tx-size 64 --max-tx-size 32
image image

Notice how the background camera noise for --max-tx-size 32 is consistently better well preserved compared to --max-tx-size 64.

Issue

Part of the SVT-AV1-PSY mainlining effort (#2269)

Author(s)

@juliobbv

Performance impact

  • quality
  • memory
  • speed
  • 8 bit
  • 10 bit
  • N/A

Test set

  • obj-1-fast can be found here
  • other
  • N/A

Merge method

  • Allow the maintainer to squash and merge when PR is ready to create a 1-commit to the master branch. The maintainer will be able to fix typos / combine commit messages to create a more readable 1-commit message or use whatever is stated in the 'Description' section
  • I will clean up my commits and the maintainer shall use 'rebase and merge' to the master branch
Edited by Julio Barba

Merge request reports

Loading