Ensure feature flags are set properly

What does this MR do?

Closes https://gitlab.com/gitlab-org/quality/nightly/issues/129

CE Port: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32923

Conformity

• [-] Changelog entry
• [-] Documentation created/updated or follow-up review issue created
• Code review guidelines
• Merge request performance guidelines
• Style guides
• [-] Database guides
• Separation of EE specific content