tests: Add tests for configurable warnings

This adds multiple tests for custom plugin warnings and core warnings,
providing checks for both cases which should cause warnings and errors
when configured as fatal.

Also adds tests for cache key calculations.