Make ClickHouse sync write buffer compatible with Redis 6
All threads resolved!
Customers using Redis 6.0 have multiple errors in the logs related to ClickHouse sync workers. These are happening because when reading from the Redis buffer to write to ClickHouse we are using lpop
with
an extra limit
argument which is not supported by Redis 6.
This MR removes the extra argument and instead uses a Redis pipeline to read the records in batches.
Customers with old redis versions should stop seeing error logs from
UsageEvents::DumpWriteBufferCronWorker
.
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
No changes to the current behavior.
asdf
this can be achieved by:
.tool_versions
of redis to 6.0.0
.tool_versions
redis to 6.0.0
PATH_TO_GDK/redis/dump.rdb
so redis 6 does not try to load the db from the newer versionasdf_install
.asdf current
and check if redis version is set to 6.0.0 on gitlab folder.rspec spec/lib/click_house/write_buffer_spec.rb
will raise failures related to compatibility problems. Another way to reproduce this error is opening a rails console and running ClickHouse::WriteBuffer.pop('anything', 1)