v1.9.0rc1 significant speed regression
In my brief testing (just 2 different clips), I've found v1.9.0rc1 commit 1dba5e72 to take about 20% more time to encode than v1.8.0 commit 59645eea . Ssimulacra2 scores are nearly identical between the two versions. I tested at --preset 2 --crf 25.
EDIT 2:
I did more testing -- increasing the scope to all presets, with 8 video samples (6 1080p 2 4k; with a majority taken from animated content). Even though I consider this issue resolved (because this change is expected), I wanted to post the results here for reference.
All tests were done via ffmpeg, with only --crf 25 and --preset X arguments. Values here are in percent difference.
average of ratios => avg(v190rc1/v180):
| name-preset | bitrate | fps | ssimu2 Mean | SSIM All | PSNR All | VMAF NEG |
|---|---|---|---|---|---|---|
| MR | 0.35% | 111.23% | -0.17% | -0.12% | -0.06% | -0.02% |
| M0 | -5.31% | -1.84% | -0.12% | -0.09% | -0.04% | -0.01% |
| M1 | -6.29% | -20.04% | -0.15% | -0.10% | -0.05% | -0.03% |
| M2 | -5.82% | -20.93% | 0.10% | 0.08% | 0.03% | 0.01% |
| M3 | -5.28% | -4.70% | 0.13% | 0.04% | 0.03% | 0.01% |
| M4 | -0.56% | 10.13% | -0.05% | -0.06% | -0.03% | 0.00% |
| M5 | 2.40% | 20.74% | 0.29% | 0.11% | 0.07% | 0.03% |
| M6 | 5.00% | 6.11% | 0.19% | 0.26% | 0.15% | 0.06% |
| M7 | 1.34% | 12.08% | -0.63% | -0.26% | -0.15% | -0.05% |
| M8 | 2.72% | 9.48% | -0.13% | -0.12% | -0.05% | -0.01% |
| M9 | 2.49% | -7.82% | 1.05% | 0.52% | 0.27% | 0.15% |
| M10 | 2.16% | 0.37% | 0.25% | 0.09% | 0.09% | 0.04% |
| M11 | 0.50% | -0.30% | 0.30% | 0.10% | 0.10% | 0.03% |
| M12 | -2.81% | -2.98% | -0.23% | 0.36% | 0.20% | -0.03% |
| M13 | 1.19% | -0.18% | 9.14% | 0.80% | 1.27% | 0.35% |
ratio of averages => avg(v190rc1)/avg(v180):
| name-preset | bitrate | fps | ssimu2 Mean | SSIM All | PSNR All | VMAF NEG |
|---|---|---|---|---|---|---|
| MR | -0.27% | 111.71% | -0.17% | -0.12% | -0.06% | -0.02% |
| M0 | -8.07% | -4.58% | -0.12% | -0.10% | -0.05% | -0.01% |
| M1 | -9.07% | -20.68% | -0.15% | -0.10% | -0.05% | -0.03% |
| M2 | -7.77% | -21.08% | 0.10% | 0.08% | 0.03% | 0.01% |
| M3 | -7.43% | -5.20% | 0.12% | 0.04% | 0.03% | 0.01% |
| M4 | -0.37% | 9.11% | -0.05% | -0.06% | -0.03% | 0.00% |
| M5 | 2.13% | 18.03% | 0.28% | 0.11% | 0.07% | 0.03% |
| M6 | 4.89% | 6.35% | 0.20% | 0.30% | 0.17% | 0.06% |
| M7 | -0.40% | 9.45% | -0.62% | -0.26% | -0.17% | -0.05% |
| M8 | 1.32% | 7.72% | -0.13% | -0.12% | -0.05% | -0.01% |
| M9 | 1.42% | -9.60% | 0.99% | 0.46% | 0.28% | 0.14% |
| M10 | 1.96% | 1.42% | 0.22% | 0.08% | 0.09% | 0.04% |
| M11 | 0.35% | 1.49% | 0.29% | 0.12% | 0.11% | 0.03% |
| M12 | -1.62% | -1.69% | -0.22% | 0.40% | 0.22% | -0.03% |
| M13 | 0.67% | 1.74% | 8.19% | 0.82% | 1.32% | 0.35% |
Note: The average of ratios and the ratio of averages are different. I don't know which one the team uses to compare performance between releases, so I posted them both. I'm also not sure which is more appropriate for this question; this serves as a showcase that they can be subtly different (e.g. did MR's bitrate increase or decrease accross the encoder change?).
Difference between presets
Click to expand
Here are the differences between presets for each release. Values are percent difference, calculated as ratio of averages.
v1.9.0rc1
| name-preset | bitrate | fps | ssimu2 Mean | SSIM All | PSNR All | VMAF NEG |
|---|---|---|---|---|---|---|
| MR-M0 | -0.68% | 187.12% | -0.42% | -0.35% | -0.17% | -0.04% |
| M0-M1 | -0.32% | 68.01% | -0.30% | -0.19% | -0.10% | -0.04% |
| M1-M2 | -0.12% | 77.21% | -0.19% | -0.10% | -0.07% | -0.03% |
| M2-M3 | 0.55% | 53.97% | -0.47% | -0.40% | -0.20% | -0.05% |
| M3-M4 | 0.04% | 61.79% | -0.45% | -0.26% | -0.17% | -0.07% |
| M4-M5 | -1.97% | 101.50% | -0.36% | -0.55% | -0.27% | -0.07% |
| M5-M6 | 0.28% | 32.31% | -0.20% | -0.23% | -0.09% | -0.02% |
| M6-M7 | -6.44% | 53.40% | -1.29% | -1.14% | -0.56% | -0.17% |
| M7-M8 | 3.82% | 29.57% | -0.18% | -0.27% | -0.10% | -0.01% |
| M8-M9 | -1.78% | 26.12% | -0.84% | -0.61% | -0.28% | -0.08% |
| M9-M10 | 2.81% | 50.73% | -1.79% | -1.76% | -0.70% | -0.08% |
| M10-M11 | 2.27% | 20.07% | -0.25% | -0.73% | -0.33% | -0.10% |
| M11-M12 | 0.11% | 8.62% | -1.59% | -0.37% | -0.27% | -0.09% |
| M12-M13 | 2.77% | 15.21% | -4.33% | -0.44% | -0.79% | -0.14% |
v1.8.0
| name-preset | bitrate | fps | ssimu2 Mean | SSIM All | PSNR All | VMAF NEG |
|---|---|---|---|---|---|---|
| MR-M0 | 7.74% | 537.05% | -0.46% | -0.37% | -0.19% | -0.05% |
| M0-M1 | 0.78% | 102.12% | -0.27% | -0.18% | -0.10% | -0.02% |
| M1-M2 | -1.53% | 78.12% | -0.43% | -0.28% | -0.15% | -0.07% |
| M2-M3 | 0.18% | 28.18% | -0.49% | -0.36% | -0.20% | -0.05% |
| M3-M4 | -7.05% | 40.56% | -0.27% | -0.16% | -0.11% | -0.06% |
| M4-M5 | -4.37% | 86.27% | -0.70% | -0.72% | -0.38% | -0.11% |
| M5-M6 | -2.36% | 46.85% | -0.12% | -0.41% | -0.18% | -0.05% |
| M6-M7 | -1.47% | 49.05% | -0.47% | -0.59% | -0.23% | -0.05% |
| M7-M8 | 2.06% | 31.66% | -0.67% | -0.41% | -0.22% | -0.06% |
| M8-M9 | -1.87% | 50.28% | -1.94% | -1.18% | -0.61% | -0.23% |
| M9-M10 | 2.27% | 34.36% | -1.03% | -1.39% | -0.51% | 0.03% |
| M10-M11 | 3.91% | 19.99% | -0.32% | -0.77% | -0.36% | -0.09% |
| M11-M12 | 2.13% | 12.12% | -1.08% | -0.64% | -0.38% | -0.03% |
| M12-M13 | 0.43% | 11.32% | -11.76% | -0.86% | -1.87% | -0.52% |
Original Results
Original Results
One sample video (Bosphorus 1080p) can be found at http://ultravideo.cs.tut.fi/video/Bosphorus_1920x1080_120fps_420_8bit_YUV_Y4M.7z \ The other is a sample from a tv-show. Both had similar increases in encode time.
Here are my test results:
1.9.0rc1 / 1.8.0 avg percent difference, 2 video samples
| preset | PSNR Y | SSIM Y | VMAF NEG | SSIMULACRA2 | bitrate | time |
|---|---|---|---|---|---|---|
| M2 | 0.0028% | 0.0007% | 0.014% | 0.124% | -0.974% | 23.001% |
svtav1 v1.9.0- 1dba5e72
❯ time ffmpeg -y -i ~/builds/025_Hunter_sample.mkv -c:v libsvtav1 -crf 25 -preset 2 /tmp/025_Hunter_sample_svtav1_c25_s2.mkv
frame= 1000 fps=5.5 q=25.0 Lsize= 12047kB time=00:00:41.79 bitrate=2361.5kbits/s speed=0.231x
ffmpeg -y -i ~/builds/025_Hunter_sample.mkv -c:v libsvtav1 -crf 25 -prese 2545.20s user 7.02s system 1409% cpu 3:01.12 total
❯ ssimulacra2_rs video -f 12 ~/builds/025_Hunter_sample.mkv /tmp/025_Hunter_sample_svtav1_c25_s2.mkv
Mean: 80.10786272 Median: 79.97899517 Std Dev: 3.67960069 5th Percentile: 74.02236606 95th Percentile: 85.81586366
size 12336283 025_full_vodes_sample_svtav1_c25_s2_v190rc1.mkv => 12336283/6239462400= 0.1977% vs raw
SSIM Y:0.987078 (18.886754) U:0.995254 (23.236494) V:0.995001 (23.011538) All:0.989761 (19.897551)
PSNR y:33.482069 u:43.691198 v:44.575828 average:35.059061 min:13.646985 max:52.711478
vmaf_v0.6.1neg.json <metric name="vmaf" min="0.000000" max="100.000000" mean="96.385378" harmonic_mean="73.844463" />
❯ time ffmpeg -y -i ~/builds/025_Hunter_sample.mkv -c:v libsvtav1 -crf 25 -preset 2 -svtav1-params tune=2 /tmp/025_Hunter_sample_svtav1_c25_s2.mkv
frame= 1000 fps=5.5 q=25.0 Lsize= 11190kB time=00:00:41.79 bitrate=2193.5kbits/s speed=0.229x
ffmpeg -y -i ~/builds/025_Hunter_sample.mkv -c:v libsvtav1 -crf 25 -prese 2527.06s user 7.23s system 1384% cpu 3:03.00 total
❯ ssimulacra2_rs video -f 12 ~/builds/025_Hunter_sample.mkv /tmp/025_Hunter_sample_svtav1_c25_s2.mkv
Mean: 79.46678378 Median: 79.29592089 Std Dev: 3.90857982 5th Percentile: 73.15151668 95th Percentile: 85.52665218
❯ time ffmpeg -y -i ~/builds/Bosphorus_1920x1080_120fps_420_8bit_YUV_Y4M/Bosphorus_1920x1080_120fps_420_8bit_YUV.y4m -c:v libsvtav1 -crf 25 -preset 2 -pix_fmt yuv420p10le /tmp/bosphorus.mkv
frame= 600 fps=4.5 q=17.0 Lsize= 7468kB time=00:00:19.96 bitrate=3064.1kbits/s speed=0.149x
ffmpeg -y -i -c:v libsvtav1 -crf 25 -preset 2 -pix_fmt yuv420p10le 1844.63s user 5.05s system 1379% cpu 2:14.13 total
❯ ssimulacra2_rs video -f 12 ~/builds/Bosphorus_1920x1080_120fps_420_8bit_YUV_Y4M/Bosphorus_1920x1080_120fps_420_8bit_YUV.y4m /tmp/bosphorus.mkv
Mean: 75.60941895 Median: 75.54633440 Std Dev: 1.35191655 5th Percentile: 73.81764962 95th Percentile: 77.63533388
size 7647415 bosphorus_svtav1_c25_s2_v190rc1.mkv => 7647415/1866243660= 0.4097% vs raw
SSIM Y:0.957858 (13.752866) U:0.983733 (17.887011) V:0.982216 (17.499744) All:0.966230 (14.714736)
PSNR y:38.818724 u:47.314764 v:46.812299 average:40.265444 min:34.574831 max:46.243371
vmaf_v0.6.1neg.json <metric name="vmaf" min="40.547837" max="95.136474" mean="80.994953" harmonic_mean="77.122051" />svtav1 v1.8.0- 59645eea
❯ time ffmpeg -y -i ~/builds/025_Hunter_sample.mkv -c:v libsvtav1 -crf 25 -preset 2 /tmp/025_Hunter_sample_svtav1_c25_s2.mkv
frame= 1000 fps=6.8 q=25.0 Lsize= 12239kB time=00:00:41.79 bitrate=2399.0kbits/s speed=0.285x
ffmpeg -y -i ~/builds/025_Hunter_sample.mkv -c:v libsvtav1 -crf 25 -prese 2090.37s user 7.50s system 1428% cpu 2:26.87 total
❯ ssimulacra2_rs video -f 12 ~/builds/025_Hunter_sample.mkv /tmp/025_Hunter_sample_svtav1_c25_s2.mkv
Mean: 80.00491319 Median: 79.93459385 Std Dev: 3.73314186 5th Percentile: 73.89112916 95th Percentile: 85.77160001
size 12532286 025_full_vodes_sample_svtav1_c25_s2_v180.mkv => 12532286/6239462400= 0.2008% vs raw
SSIM Y:0.987075 (18.885604) U:0.995123 (23.118188) V:0.994888 (22.914026) All:0.989718 (19.879331)
PSNR y:33.481664 u:43.670990 v:44.554224 average:35.057807 min:13.646606 max:52.700196
vmaf_v0.6.1neg.json <metric name="vmaf" min="0.000000" max="100.000000" mean="96.383620" harmonic_mean="73.843725" />
❯ time ffmpeg -y -i ~/builds/025_Hunter_sample.mkv -c:v libsvtav1 -crf 25 -preset 2 -svtav1-params tune=2 /tmp/025_Hunter_sample_svtav1_c25_s2.mkv
frame= 1000 fps=6.7 q=25.0 Lsize= 11577kB time=00:00:41.79 bitrate=2269.3kbits/s speed=0.278x
ffmpeg -y -i ~/builds/025_Hunter_sample.mkv -c:v libsvtav1 -crf 25 -prese 2125.60s user 7.06s system 1418% cpu 2:30.35 total
❯ ssimulacra2_rs video -f 12 ~/builds/025_Hunter_sample.mkv /tmp/025_Hunter_sample_svtav1_c25_s2.mkv
Mean: 79.34420999 Median: 79.18981585 Std Dev: 3.98580841 5th Percentile: 72.92725466 95th Percentile: 85.54812216
❯ time ffmpeg -y -i ~/builds/Bosphorus_1920x1080_120fps_420_8bit_YUV_Y4M/Bosphorus_1920x1080_120fps_420_8bit_YUV.y4m -c:v libsvtav1 -crf 25 -preset 2 -pix_fmt yuv420p10le /tmp/bosphorus.mkv
frame= 600 fps=5.5 q=17.0 Lsize= 7497kB time=00:00:19.96 bitrate=3075.9kbits/s speed=0.183x
ffmpeg -y -i -c:v libsvtav1 -crf 25 -preset 2 -pix_fmt yuv420p10le 1535.20s user 5.13s system 1408% cpu 1:49.33 total
❯ ssimulacra2_rs video -f 12 ~/builds/Bosphorus_1920x1080_120fps_420_8bit_YUV_Y4M/Bosphorus_1920x1080_120fps_420_8bit_YUV.y4m /tmp/bosphorus.mkv
Mean: 75.51960672 Median: 75.44787275 Std Dev: 1.35956509 5th Percentile: 73.74462705 95th Percentile: 77.57958921
size 7677018 bosphorus_svtav1_c25_s2_v180.mkv => 7677018/1866243660= 0.4113% vs raw
SSIM Y:0.957846 (13.751611) U:0.983551 (17.838705) V:0.982098 (17.470936) All:0.966172 (14.707261)
PSNR y:38.816943 u:47.265028 v:46.785197 average:40.261141 min:34.581210 max:46.243966
vmaf_v0.6.1neg.json <metric name="vmaf" min="40.639329" max="95.143227" mean="80.973702" harmonic_mean="77.105095" />Bosphorus encode time v1.9.0rc1 / v1.8.0 => 134.13 / 109.33 = 1.227
EDIT:
Added file size, psnr, ssim, and vmaf scores
System: CPU Intel i7-12700k power limit 70W ; RAM 32GB DDR4 3600MHz
Test results data: svtav1_test_results_v180-v190rc1.tsv
Edited by Wombat