GitLab

Created by: szarsti

We run ddb with multiple buckets in different resolution.

Whenever ddb receives new point to unknown bucket, it will create one with default resolution and number of points per file.

This behaviour has caused us some serious problems on our production. When we were adding new box to cluster we used automated provisioning which opened incoming traffic to newly started node. I imagine that it started receiving data, before it fully synchronised all buckets meta-data. AS a result it just created buckets with default parameters and even some of those new values got populated to other nodes in a cluster. We ended up with in a situation that some mstores were created with 1sec resolution, other with 10s resolution for the same bucket.

This whole situation was very difficult to diagnose and took a lot of effort to restore. We had to delete some mstores that were created with wrong resolution, potentially loosing some data.

I am proposing to make ddb log error and not accept data for unknown buckets. I think this will make it less prone to error due to not so careful management.

I have also noticed, that one can run command line utility to create the same bucket multiple times with different resolution. Calling it that way effectively acts as altering bucket meta-data. I think that should be blocked as well, because if one truly wants to change bucket resolution, they should make sure that all mstores get converted. Otherwise node will run into errors on next restart.

If you think that this is too much change, breaking backwards compatibility, maybe we should make it driven by some configuration switch. We would definitely want to run ddb with that setting enabled.