Remove unnecessary config struct pointers NO CHANGELOG
While reviewing !640 (merged) I noticed that we have some sections in the configuration struct that use pointers without needing them. This is inconsistent with how we manage other config fields and it leads to unnecessary if cfg.Foo == nil
checks.