undefined | Better HN

0 pointsidunno24612y ago0 comments

Pushing a couple terabytes a day through kafka 0.7. We don't use zookeeper on the producing side and it alleviates this a lot. It's a little more brittle pushing host/partition configs around, but we accepted loss of data in this system and its worth the simplicity of it. Also played with the idea of putting an elb in front.

I'm having way more trouble with the consumer being dumb with the way it distributes topics and partitions. End up with lots of idle consumers, while others are way above max.

0 comments

mikebabineau12y ago

Thanks for the note, we'll have to take a look at that sort of configuration.

Your consumer problems sounds similar to one we had. Root cause was that the number of consumers exceeded the number of active partitions. The tricky part was that the topic was only distributed across part of the cluster (because of the issue described in my parent post), so we had fewer partitions than we thought.

j / k navigate · click thread line to collapse

0 comments

mikebabineau12y ago

Thanks for the note, we'll have to take a look at that sort of configuration.

j / k navigate · click thread line to collapse