Skip to content

Feature/create kafka topics

Joe Wass requested to merge feature/create-kafka-topics into develop

feat: Kafka topic creation and logging

Auto-topic-creation doesn't seem to work reliably with Labda-based producers. There's a race condition between the lambda function wanting to run and waiting for the initial creation of the topic.

As the topics are consumed primarily by the Cayenne indexer it's reasonable to ask it to create the topics on startup. This is a similar pattern to how it creates Elastic Search indexes with relevant mappings on startup.

Cayenne creates topics using configuration values in the source code. It should be triggered at start-up time in the same way that indexes are.

Each relevant topic, i.e. "citation" and "xml" should be created.

Also we need to log partition lag better. In order to discover if a Cayenne indexer has got too far behind, it should log its lag. This log should be structured, allowing it to be picked up by a CloudWatch alarm if necessary.

issues#881 (closed) issues#880 (closed)

Edited by Joe Wass

Merge request reports