undefined | Better HN

0 pointsteraflop10y ago0 comments

As the article explains at length, using Zookeeper or etcd as a "locking service" in this way is not safe, even if the service is perfect and failure-free. There is a fundamental race condition between finding out that you've obtained a lock, and doing some other operation that assumes mutual exclusion.

0 comments

bkeroack10y ago

It's not really "fundamental". It's simply that the process that acquires the lock can fail (or be paused, or partitioned away from everything else, etc), and if it does, but then comes back later with a valid lock, bad things may happen.

The author's solution is to push serialization logic into the resource/storage layer (by checking fencing tokens). But what if the resource is itself distributed? Then it needs it's own synchronization mechanism? It's locks all the way down.

bkeroack10y ago

Thinking more about it, this is a fundamental weakness of having self-policing processes, which I suppose is the OP's main point. It can be mitigated by having infinite lock TTLs, at the cost of risking system deadlock on process failure. Thank you to GPP for spurring me to think more deeply about this.

As I stated, though, if the resource being protected is either a distributed system itself, or a system that cannot support fencing logic, this failure mode is difficult or impossible to prevent. The frequency of failure should be kept in mind here: most services can probably guarantee 99.99% uptime against the likelihood of 5 minute GC pauses.

j / k navigate · click thread line to collapse

0 pointsteraflop10y ago0 comments

0 comments

bkeroack10y ago

j / k navigate · click thread line to collapse