Cayenne should create required Kafka topics on startup
Background
Auto-topic-creation doesn't seem to work reliably with Labda-based producers. There's a race condition between the lambda function wanting to run and waiting for the initial creation of the topic.
As the topics are consumed primarily by the Cayenne indexer it's reasonable to ask it to create the topics on startup. This is a similar pattern to how it creates Elastic Search indexes with relevant mappings on startup.
Expected behavior
Cayenne creates topics using configuration values in the source code. It should be triggered at start-up time in the same way that indexes are.
Each relevant topic, i.e. "citation" and "xml" should be created.
Definition of done
-
Unit tests identified, implemented, and passing -
Code reviewed -
Available via a staging URL -
Knowledge base reviewed and updated -
Public documentation reviewed and updated -
Consider any impacts to current or future architecture/infrastructure, and update specifications and documentation as needed -
Acceptance criteria met -
New profile create-kafka-topics
added to code. -
Deployment container config should add create-kafka-topics
in the same places we havecreate-mappings
. -
An empty control message should be sent on startup, which will trigger Kafka into creating the topic. This runs with no change if the topic already exists. -
The Kafka ingester ignores this control message when it's consuming.
-
Notes
Edited by Dominika Tkaczyk