Post-processing normals are too imprecise for reflection effects
Right now normals are passed to post-processing shaders as 8:8:8 x:y:z integers, like normal map textures. This isn't accurate enough for reflections, and will cause plain, visible banding. (See "naive" image in https://imgur.com/a/x5TvUCS)
Either OpenMW should use an rgb16 render target (unlikely), or it should pack normals in a planar format, as 12:12.
The current most widely used planar normal formats are octagonal and hemi/semi-octagonal packing, where the normal sphere is crunched into an octagon, then projected onto a plane with the overlapping parts of the projection moved around to not overlap. We can't use hemi/semi-octagonal packing. Octahedral packing involves no trig (Except for a call to normalize() at the end of decoding) A slow implementation full of branches, which also packs the planar coordinates into a vec3 meant for a 24-bit render target, looks like as follows.
// Some parts inspired by other implementations, but only non-copyrightable trivial parts
// Original elements under the CC0: https://creativecommons.org/publicdomain/zero/1.0/legalcode
vec3 normal_encode(vec3 normal)
{
// map onto octagon
normal *= 1.0/(abs(normal.x) + abs(normal.y) + abs(normal.z));
// project
float n = normal.x;
float m = normal.y;
// shift projection for back layer
if (normal.z < 0)
{
n += 0.5;
m += 0.5;
if (n > 1.0)
n -= 2.0;
if (m > 1.0)
m -= 2.0;
}
uint ni = uint(round(n * 2047.0) + 2047.0);
uint mi = uint(round(m * 2047.0) + 2047.0);
return vec3(float(ni >> 4),
float(((ni & uint(0xF)) << 4) | (mi >> 8)),
float(mi & uint(0xFF))
)/255.0;
}
vec3 normal_decode(vec3 normal)
{
uint ix = uint(round(normal.x * 255.0));
uint iy = uint(round(normal.y * 255.0));
uint iz = uint(round(normal.z * 255.0));
uint ni = (ix << 4) | ((iy >> 4) & uint(0xF));
uint mi = iz | ((iy & uint(0xF)) << 8);
float n = (float(ni) - 2047.0) / 2047.0;
float m = (float(mi) - 2047.0) / 2047.0;
// unproject front
float x = n;
float y = m;
float z = 0.0;
// shift x/y and flip z if back layer
if (abs(n) + abs(m) > 1.0)
{
x -= 0.5;
y -= 0.5;
if (x < -1.0)
x += 2.0;
if (y < -1.0)
y += 2.0;
z = -(1.0 - (abs(x) + abs(y)));
}
else
{
z = 1.0 - (abs(x) + abs(y));
}
return normalize(vec3(x, y, z));
}
Octahedral packing can be implemented without packing. This implementation is designed to demonstrate how it algorithmically works.
Alternative methods include: store x and y and z's sign then reconstruct z (has more banding than other approaches, see https://imgur.com/a/BgdPQSz); spherical coordinates (involves trig, has higher storage density towards the poles), spiral packing (weird and disused).
The main downside to octahedral packing is that it interpolates poorly and has discontinuities, but the normal map is already aliased (as it should be), so that's not an issue for this use case. Also, any planar packing would have discontinuities in at least one direction, because you can't project the surface of a sphere (or ANY polyhedron) to something with the topology of a torus (planar coordinates); for example, spherical coordinates only wrap horizontally, and are discontinuous between the top and bottom.