CONFIG_QM_EXTENSION: Question on valid range of quantizer deltas

CONFIG_QM_EXTENSION introduces a way of specifying a quantizer matrix.

What constraints should I put in the specification for the valid range of delta values?

At the moment there is only one constraint excluding a very large value:

      int16_t prev = 32;
      for (int i = 0; i < tx_size_2d[tsize]; i++) {
        const int32_t delta = aom_rb_read_svlc(rb);
        if (delta == INT32_MIN) {
          aom_internal_error(error_info, AOM_CODEC_CORRUPT_FRAME,
                             "Invalid matrix_coef_delta");
        }
        prev = (prev + delta + NUM_QM_VALS) % NUM_QM_VALS;
        mat[s->scan[i]] = prev;
      }

However, the reference code has problems for some other values.

In particular, NUM_QM_VALS is equal to 256, so if prev = 0, delta = -300, then the quantizer value stored will be negative.

Also, if a value of 0 is chosen for the quantizer, an assert fires (in debug mode) or a divide by zero exception happens (in release mode):

static void calc_wt_matrix(const int txsize, const qm_val_t *iwt_matrix,
                           qm_val_t *wt_matrix) {
  const int size = tx_size_2d[txsize];
  for (int i = 0; i < size; ++i) {
    assert(iwt_matrix[i] != 0);
    wt_matrix[i] = (int)1024 / iwt_matrix[i];
  }
}

Proponents

ximin.zhang chuangh @PDX-Runner @juliobbv maryla @wantehchang