Assertion failure in av1_decide_states_avx2

What version / commit were you testing with?

1d5c1754

What steps will reproduce the problem?

  1. Build AVM with assertions on:

    cmake ../avm -DENABLE_DOCS=0 -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_ASSERTS=1 && make  -j64
  2. Run this aomenc command (CTC RA with 17 frames)

    ./aomenc ${HOME}/videos/av2ctc/a5_270p/SparksElevator_480x270p_5994_10bit.y4m  --tile-columns=0 --threads=1 --lag-in-frames=19 --auto-alt-ref=1 --min-gf-interval=16 --max-gf-interval=16 --gf-min-pyr-height=4 --gf-max-pyr-height=4 --kf-min-dist=65 --kf-max-dist=65 --use-fixed-qp-offsets=1 --enable-intrabc-ext=2  --test-decode=warn --cpu-used=0 --passes=1 --end-usage=q --deltaq-mode=0 --enable-tpl-model=0 --enable-keyframe-filtering=0 --obu --psnr -v  --qp=235 --skip=0 --limit=17 -o /dev/null

What is the expected output?

Success

What do you see instead?

baseline_gf_interval = 16
POC:     0 [ KEY ][BRU0:-1][Level:1][Q:238]:        623 Bytes, 764346.0ms, 27.2350 dB(Y), 39.8317 dB(U), 35.6731 dB(V), 28.7858 dB(Avg)    [ -1, -1, -1, -1, -1, -1, -1,]
POC:    16 [INTER][BRU0:-1][Level:1][Q:254]:        103 Bytes, 167717.1ms, 26.5752 dB(Y), 39.5305 dB(U), 35.2778 dB(V), 28.1393 dB(Avg)    [  0,  0,  0,  0,  0,  0,  0,]
POC:     8 [INTER][BRU0:-1][Level:2][Q:271]:         34 Bytes, 42421.5ms, 26.6032 dB(Y), 39.7025 dB(U), 35.5752 dB(V), 28.1774 dB(Avg)    [  0, 16,  0,  0,  0,  0,  0,]
POC:     4 [INTER][BRU0:-1][Level:3][Q:255]:          8 Bytes, 35875.0ms, 26.5391 dB(Y), 39.7604 dB(U), 35.6158 dB(V), 28.1179 dB(Avg)    [  0,  8, 16,  0,  0,  0,  0,]
POC:     2 [INTER][BRU0:-1][Level:4][Q:247]:          8 Bytes, 27716.7ms, 26.7820 dB(Y), 39.7981 dB(U), 35.6412 dB(V), 28.3518 dB(Avg)    [  0,  4,  8, 16,  0,  0,  0,]
aomenc: /tmp/avm/av1/encoder/x86/trellis_quant_avx2.c:73: void av1_decide_states_avx2(const struct tcq_node_t *, const struct tcq_rate_t *, const struct prequant_t *, int, int, int64_t, struct tcq_node_t *): Assertion `(rdmult >> 32) == 0' failed.

Please use labels and text to provide additional information.