undefined | Better HN

0 pointsphilsnow2y ago0 comments

The diagram on https://virtual-kubelet.io/docs/architecture/ makes me wonder whether it's possible to have a k8s cluster where the nodes are all virtual kubelets backed by different cloud providers (and then perhaps schedule loads preferentially with selectors)

0 comments

kuhsaft2y ago

I think it’s completely possible. Though, you’ll have to manage your own control-plane.

Azure AKS and EKS provide virtual-kubelet functionality in some form, but AKS is an a managed control-plane where you can’t add nodes yourself and EKS only allows nodes in the same VPC.

Edit: It already is a thing. https://github.com/virtual-kubelet/tensile-kube

philsnowOP2y ago

tensile-kube seems to be structured as a "k8s cluster of k8s clusters", with an upper kubemaster farming out resources to lower kubemasters (through virtual-node). I don't know if there's any particular reason to have that separation; possibly the lower kubemasters could be removed and you could just run a bunch of virtual-kubelets.

kuhsaft2y ago

I think the biggest hurdle would be networking between the pods since they will be running on different cloud providers.

philsnowOP2y ago

I've seen some people using wireguard for intra-cluster networking so that all their nodes can run pretty much anywhere.

harpratap2y ago

Wouldn't the network cost be absurd in such case? Not only the pod-to-pod communication cost skyrocket, all the heartbeats, health checks, metrics, daemonsets pinging each other will probably end up costing more than the CPU and Memory

j / k navigate · click thread line to collapse

0 comments

kuhsaft2y ago

I think it’s completely possible. Though, you’ll have to manage your own control-plane.

Azure AKS and EKS provide virtual-kubelet functionality in some form, but AKS is an a managed control-plane where you can’t add nodes yourself and EKS only allows nodes in the same VPC.

Edit: It already is a thing. https://github.com/virtual-kubelet/tensile-kube

philsnowOP2y ago

kuhsaft2y ago

I think the biggest hurdle would be networking between the pods since they will be running on different cloud providers.

philsnowOP2y ago

I've seen some people using wireguard for intra-cluster networking so that all their nodes can run pretty much anywhere.

harpratap2y ago

j / k navigate · click thread line to collapse