From wiki: > Since the hash function is randomizing, each of the n sites is equally likely to receive the object O. Loads are uniform across the sites.
This is a really interesting topic of its own! The first sentence is correct, the second is asymptotically correct, but untrue in a lot of practical cases. It's very easy to overestimate the degree of uniformity you get from random allocation, especially where the number of "balls" isn't very large compared to the number of "bins". More here: https://brooker.co.za/blog/2018/01/01/balls-into-bins.html
For error handling when the task fails it sends a message to the genserver we can also use that opportunity to handle retry, or do a strategy change where after first failure we throw it into a job queue.
This way we optimize for user experience and at the same time have a robust strategy for handling failure. I guess that can be another blog post on its own.
1. While JCH will usually be the most performant hashing method, naively, removing a node will affect all nodes of higher order. This makes the logic of node deletions somewhat more complex than (say) Discord's hash ring. This is why JCH is more common for long-term, distributed, redundant storage -- where the topology changes far less frequently.
2. For sharding, what makes distribution hard is not so much the hashing but consensus on the cluster state -- this is the hidden problem. Bryan Hunter's talk on Waterpark (https://youtu.be/9qUfX3XFi_4) is a excellent example of what you can do when you can set things up so that the topology is fixed. In fact, this approach makes things so straight forward that it is shared by Riak, where the number of vnodes is fixed.
However, if you have a rapidly changing topology (like several Kubernetes clusters that are frequently scaling up and down), you can often need some sort of consensus mechanism to make sure every node has a consistent view of the cluster. In my experience, this usually ends up being the most complex part of distribution problem to solve.
Thank you for your feedback.
