undefined | Better HN

0 pointstaf212y ago0 comments

I would definitely take slower write speed to have this extra assurance of high availability with data consistency... as it is today - it's pretty hard operate redis in a HA way even with sentinel... I'm sure others have figured it out but for me the best I have is a 30 second window when sentinel is switching the backup slave to master and either writes are lost haproxy can hold but it works sometimes... I can imagine with synchronous replication we'd have slower write speed but at least failover could be sub-second and possibly zero downtime... using a vip/keepalived - Still redis is such a good datastore, it's difficult to imagine a world/service without it. for what it's worth this looks to be the best setup i've found: http://failshell.io/sensu/high-availability-sensu/ would be interested if others have other setups that possibly work better...

0 comments

antirez12y ago

Hey, going a bit off topic about Sentinel, this should not happen. Sentinel + Redis + Clients as a distributed system is not able to guarantee strong consistency with arbitrary partitions clearly, but in the scenario you describe, that is, the master crashes or is partitioned away alone, the window should be like a fraction of a millisecond (the replication link latency) most of the times.

EDIT: to clarify:

1) Sentinels will tell clients master is A.

2) A fails.

3) Sentinels will still tell it is A before failover.

4) Failover happens.

5) Sentinels will finally tell master is B.

taf2OP12y ago

could be something that haproxy is causing - when switching masters... maybe I need a script that handles switching the vip when sentinel detects failure in master instead of haproxy with a backup... btw - thank you for redis.

seiji12y ago

You shouldn't need haproxy in the loop at all.

If your redis client is Sentinel-aware, you ask your client directly "give me the redis master server" and it asks Sentinel which server is master. The only ip address involved for your configuration is the address of the sentinel(s).

haproxy will take 30 seconds to detect your failure. Sentiel will notify you immediately. Sentinel is the one determining the exact state of the master election, and it'll be very chatty when a new master replaces the old one.

j / k navigate · click thread line to collapse