undefined | Better HN

0 pointsopportune7y ago0 comments

From my experience, Kafka has the best api for handling read-once, distributed streams. Almost every other streaming solution, like Redis in this case, has a non-ideal or non-existent way to coordinate stream consumers in a way that prevents double-reads. And lots of streaming applications need to ensure read-once (think about what a double read ends up as - maybe a twice-sent message, or a duplicate metric), so I'm not sure why they all struggle so much with just copying kafka's pretty simple consumer api

0 comments

manigandham7y ago

How so? Kafka only accepts offsets which are meant for batches of items or even an entire partition. This means a single item not being processed within that batch requires your own code to compensate. It's the weakest of all the messaging models.

Per-message acknowledgement is an advancement. Redis requires manual lookups for unacknowledged items but you can also use Apache Pulsar for a more scalable distributed disk-based system which itself is a solid evolution over Kafka's design.

Also note that "exactly-once" semantics are actually impossible. Messaging systems are either "at-least-once" or "at-most-once". Kafka has some attempts at using transactions to solve this but that's only when using Kafka streams and only ensures read progress, not the processing result.

aarbor9897y ago

I’m not sure what you mean by “only accepts offsets which are meant for batches” but with Kafka the offsets are per-partition and you have to flexibility to control exactly when that offset is marked as processed. In our systems we always used manual offset committing and would only commit an offset once processing of the message has completed successfully to ensure both at-most-once and seamless failover.

manigandham7y ago

Offsets are a marker that says everything before it (in that partition) is processed. This creates 2 issues:

1) Your application must coordinate and make sure that everything up to that offset is indeed processed successfully. 2) You application must stop if it encounters an error (because it can't commit an offset greater than that item) or handle it separately by logging to another topic, database, etc.

Other systems like Redis, Pulsar, Google PubSub provide per-message acknowledgement to allow items to be individually processed without blocking other forward progress.

1 more reply

senderista7y ago

That doesn’t sound like “at-most-once”. What if your consumer crashed after processing the message but before committing the offset?

1 more reply

j / k navigate · click thread line to collapse

0 comments

manigandham7y ago

aarbor9897y ago

manigandham7y ago

Offsets are a marker that says everything before it (in that partition) is processed. This creates 2 issues:

Other systems like Redis, Pulsar, Google PubSub provide per-message acknowledgement to allow items to be individually processed without blocking other forward progress.

1 more reply

senderista7y ago

That doesn’t sound like “at-most-once”. What if your consumer crashed after processing the message but before committing the offset?

1 more reply

j / k navigate · click thread line to collapse