Support unique topics for kafka_consumer fixture
I noticed recently in my tests that starting/stopping Zookeeper & Kafka came at a performance cost, and if instead I scoped them to the session I could save 2 minutes on running tests.
The issue is each test would still need unique topics, however make_kafka_consumer
wants the topics to be known up-front in order to use the seek_to_beginning
feature which is quite useful.
For my own project I was able to work-around this by calling make_kafka_consumer
without topics, and then made my own fixture mimicking the seek_to_beginning
functionality, but I was wondering if you thought it could be something wired directly into pytest-kafka itself?
code looks something like:
_kafka_consumer = pytest_kafka.make_kafka_consumer('kafka_server', scope='function')
@pytest.fixture(name='kafka_topic', scope='function')
def kafka_topic_fixture():
yield f'unittest_{time.time()}'
@pytest.fixture(name='kafka_consumer', scope='function')
def kafka_consumer_fixture(kafka_topic, _kafka_consumer):
_kafka_consumer.subscribe([kafka_topic])
# Wait until we have assigned partitions
start = time.time()
end = start + PARTITION_ASSIGNMENT_TIMEOUT
partitions_assigned = False
while not partitions_assigned and time.time() <= end:
_kafka_consumer.poll(timeout_ms=20)
partitions_assigned = len(_kafka_consumer.assignment()) > 0
if not partitions_assigned:
time.sleep(0.1)
assert partitions_assigned
_kafka_consumer.seek_to_beginning()
yield _kafka_consumer