Kafka Streams crash when encountering Axual Offset Distribution Metadata

Summary

When working with an Axual installation with more than one cluster the Offset Distribution can cause Kafka Streams apps to fail when switching between clusters. Confirmed issue on Client 5.5.0, 5.5.2, 5.5.3, 5.7.1, 5.7.2, 5.7.3 and 5.8.0

Steps to reproduce

1. Start and run a streams app on one cluster in an Axual Instance containing more clusters.
2. Stop the Streams app and wait for distribution to catch up
3. Start the Streams app connected to the other cluster. The failure occurs, see log

Example Project

N/A

What is the current bug behavior?

Whenever Axual Streams encounters the Offset distribution metadata it throughs an example

What is the expected correct behavior?

Whenever the Offset distribution metadata is found, it should be cleaned for the Streams engine to parse

Relevant logs and/or screenshots

2021-11-16 12:56:36.020  INFO 25 --- [-StreamThread-2] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=example-app-3, groupId=axual-demo-example-example-app] Unsubscribed all topics or patterns and assigned partitions
	at java.base/java.util.Base64$Decoder.decode(Unknown Source) ~[na:na]
	at org.apache.kafka.streams.processor.internals.StreamTask.decodeTimestamp(StreamTask.java:1039) ~[kafka-streams-2.7.0.jar:na]
	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) ~[kafka-streams-2.7.0.jar:na]
	at org.apache.kafka.streams.processor.internals.StreamTask.initializeTaskTime(StreamTask.java:837) ~[kafka-streams-2.7.0.jar:na]
java.lang.IllegalArgumentException: Illegal base64 character 7b
	at org.apache.kafka.streams.processor.internals.StreamThread.initializeAndRestorePhase(StreamThread.java:728) ~[kafka-streams-2.7.0.jar:na]
	at org.apache.kafka.streams.processor.internals.StreamTask.initializeMetadata(StreamTask.java:818) ~[kafka-streams-2.7.0.jar:na]
2021-11-16 12:56:36.127 ERROR 25 --- [-StreamThread-1] org.apache.kafka.streams.KafkaStreams    : stream-client [streams-example-app-1] All stream threads have died. The instance will be in error state and should be closed.
	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) ~[kafka-streams-2.7.0.jar:na]
java.lang.IllegalArgumentException: Illegal base64 character 7b

Possible fixes

Clean the OffsetAndMetadata objects in the proxy clients to clean the distribution metadata. Classes to clean: AdminClient ListConsumerGroupOffsets and Consumer committed methods