undefined | Better HN

0 pointsaeyes3y ago0 comments

Most people only use etcd at small scale. If you try to store 10 or even 100GB in etcd you are going to run into uncommon problems.

Most people don't even know that the Kubernetes control plane by default has a hard limit on etcd size. It used to be 2GB, not sure what it is now.

0 comments

mdaniel3y ago

I would need a citation on the kubernetes control plane having any such hard limit. etcd is its own little snowflake, and I could very easily imagine it having some bad default value like that, or even kubeadm improperly configuring it

However, related to that, for big-time clusters (q.v. https://news.ycombinator.com/item?id=35174655 and https://news.ycombinator.com/item?id=25907312) one should without question move events over into their own etcd cluster: https://openai.com/research/scaling-kubernetes-to-2500-nodes...

dilyevsky3y ago

It’s etcd default https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limi...

There’s also max object size of 1MB on the apiserserver side I believe

ivzhh3y ago

ByteDance replaces etcd with kubebrain [1], which is backed by their own KV store (TiKV seems also supported).

The single-group raft is the hard limit.

[1]: https://github.com/kubewharf/kubebrain

mdaniel3y ago

That's interesting, thanks for the link. I held out high hopes for pluggable KV in kubernetes for the longest time, but since that issue was closed WONTFIX I resigned my hopes

Heh, that kubebrain TODO is some "oh, really?"

* Guarantee consistence in critical cases

but I give them huge props for calling out Jepsen

kbumsik3y ago

Doesn't Consul have the similar storage limit btw?

I have seen very few strongly consistent distributed KV store that scales beyond 10GB+

dilyevsky3y ago

It’s the underlying db limit (boltdb) which both etcd and consul use

dilyevsky3y ago

I’ve run it with over 50G under heavy load (10k+ qps) and it was fine. It’s pretty sensitive to disk latency though

j / k navigate · click thread line to collapse

0 comments

mdaniel3y ago

dilyevsky3y ago

It’s etcd default https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limi...

There’s also max object size of 1MB on the apiserserver side I believe

ivzhh3y ago

ByteDance replaces etcd with kubebrain [1], which is backed by their own KV store (TiKV seems also supported).

The single-group raft is the hard limit.

[1]: https://github.com/kubewharf/kubebrain

mdaniel3y ago

That's interesting, thanks for the link. I held out high hopes for pluggable KV in kubernetes for the longest time, but since that issue was closed WONTFIX I resigned my hopes

Heh, that kubebrain TODO is some "oh, really?"

* Guarantee consistence in critical cases

but I give them huge props for calling out Jepsen

kbumsik3y ago

Doesn't Consul have the similar storage limit btw?

I have seen very few strongly consistent distributed KV store that scales beyond 10GB+

dilyevsky3y ago

It’s the underlying db limit (boltdb) which both etcd and consul use

dilyevsky3y ago

I’ve run it with over 50G under heavy load (10k+ qps) and it was fine. It’s pretty sensitive to disk latency though

j / k navigate · click thread line to collapse