Right now, shadows just use the 2x2 PCF filter provided by
sampler2DShadow, so they're fairly hard and sometimes pixelated. Shadows aren't always hard in real life, and are very, very rarely pixelated.
Possible approaches include:
- Using a bigger PCF (Percentage Closer Filtering) filter. GLSL 1.2 doesn't offer any especially nice ways of doing this, and using a bigger filter might cause shadow acne issues. This would also soften shadows that would be hard in real life.
- Using a statistical shadow sampling approach, such as Variance Shadow Maps or Exponential Shadow Maps. These allow standard texture filtering techniques to work with shadow maps, so can make shadows soft without as much of a performance hit as PCF, but need tuning to avoid artefacts like light bleed. Also, softening affects everything, not just shadows that need softening.
- Using PCSS (Percentage-Closer Soft Shadows). This allows soft shadows where they'd be soft in real life and hard shadows where they'd be hard. However, it's likely to be much slower - the implementation suggested in the original paper does 100 shadow map lookups per fragment instead of the four we're doing now. It's also likely to need tweaking to still be accurate when combined with LiSPSM, which OpenMW uses.
- Using a hybrid of a statistical approach and PCSS. I've seen evidence of it being done, and it's not as slow as PCSS of the same quality, but looks a lot harder to implement.