Performance test for v1

A few changes happened in the 1.0 epic:

• Store more data in the BufferedItem.
• Slice packages differently.
• Some logging changes.
• Partition support.

It would be nice to profile the Siphon producer again and see if we have bottlenecks (golang).

Perf testing with many tables

To prepare for handling more tables, we should also profile Siphon with 500 tables (approximate number of tables in the main DB).

Most of the tables would get very little to 0 traffic, so test accordingly:

Idea:

Write a script that generates 500 tables (simple int columns only) and the producer/consumer configuration.

Write another script to generate data (initial data snapshot can be skipped) using a simple distribution function where about 20% of the tables gets most of the traffic (90%).

Expectation:

We expect that the memory buffer, serialization and compression would use significant runtime and resources. Depending on the results we might need to look into parallelization.