CONFIG_TX_PARTITION_TYPE_EXT: Incorrect transform type from av1_get_tx_type

How was the issue detected?

HDR Nova AV2 Verification

What version / commit were you testing with?

c42b3607

What steps will reproduce the problem?

We are making verification streams for CONFIG_TX_PARTITION_TYPE_EXT, but have a concern about an optimization in the reference code that is no longer valid.

The concern is in av1_read_coeffs_txb_facade:

   if (plane == 0) {
      const int txw = tx_size_wide_unit[tx_size];
      const int txh = tx_size_high_unit[tx_size];
      // The 16x16 unit is due to the constraint from tx_64x64 which sets the
      // maximum tx size for chroma as 32x32. Coupled with 4x1 transform block
      // size, the constraint takes effect in 32x16 / 16x32 size too. To solve
      // the intricacy, cover all the 16x16 units inside a 64 level transform.
      if (txw == tx_size_wide_unit[TX_64X64] ||
          txh == tx_size_high_unit[TX_64X64]) {
        const int tx_unit = tx_size_wide_unit[TX_16X16];
        const int stride = xd->tx_type_map_stride;
        for (int idy = 0; idy < txh; idy += tx_unit) {
          for (int idx = 0; idx < txw; idx += tx_unit) {
#if CONFIG_INTER_IST
            xd->tx_type_map[(row + idy) * stride + col + idx] = tx_type1;
#else
            xd->tx_type_map[(row + idy) * stride + col + idx] = tx_type;
#endif  // CONFIG_INTER_IST
          }
        }
      }
    }

The problem occurs with the new transform partition when the block is divided into 32x16 and 32x32 and 32x16 blocks. In this case, the condition does not trigger (as neither dimension is 64), but the copying is still required.

Fix

Deleting the condition is enough to get correct operation:

   if (plane == 0) {
      const int txw = tx_size_wide_unit[tx_size];
      const int txh = tx_size_high_unit[tx_size];
      // The 16x16 unit is due to the constraint from tx_64x64 which sets the
      // maximum tx size for chroma as 32x32. Coupled with 4x1 transform block
      // size, the constraint takes effect in 32x16 / 16x32 size too. To solve
      // the intricacy, cover all the 16x16 units inside a 64 level transform.
      {  // MODIFIED LINE
        const int tx_unit = tx_size_wide_unit[TX_16X16];
        const int stride = xd->tx_type_map_stride;
        for (int idy = 0; idy < txh; idy += tx_unit) {
          for (int idx = 0; idx < txw; idx += tx_unit) {
#if CONFIG_INTER_IST
            xd->tx_type_map[(row + idy) * stride + col + idx] = tx_type1;
#else
            xd->tx_type_map[(row + idy) * stride + col + idx] = tx_type;
#endif  // CONFIG_INTER_IST
          }
        }
      }
    }