Fix chroma artifacts in high density presets

Description

Fix the atb check in the encode pass to separate the 10-bit and 8bit paths rather than atb_mode paths

Issues

Closes #400 (closed) Ref #424 (closed)