undefined | Better HN

0 pointsherval5y ago0 comments

Have you tried or considered Nomad (from the makers of Terraform)?

0 comments

I haven't, and since I've sunk the cost into Kubernetes and know it very well now, likely won't end up there.

In retrospect though, maybe it's exactly what I needed. Great suggestion.

I've been using Nomad for my "toy" network, and I like it. It runs many services, and a few periodic jobs. Lightweight, easy to set up, and has enough depth to handle some of the weirder stuff.

q3k5y ago

Nomad, in its free offering, cannot compete with k8s for organization-wide usage:

- no RBAC

- no quotas

- no preemption

- no namespacing

This means: everyone is root on the cluster, including any CI/CD system that wants to test/update code. And there's no way to contain runaway processes with quotas/preemption.

p_l5y ago

can it handle networking (including load balancing and reverse proxies with automatic TLS) or virtualized persistent storage? Make it easy to integrate common logging system?

Cause those are the parts that I miss probably the most when dealing with non-k8s deployment, and I haven't had the occasion to use Nomad.

adadgar5y ago

For load balancing you can just run one of the common LB solutions (nginx, haproxy, Traefik) and pick up the services from the Consul service catalog. Traefik makes it quite nice since it integrates with LetsEncrypt and you can setup the routing with tags in your Nomad jobs: https://learn.hashicorp.com/nomad/load-balancing/traefik

What Nomad doesn’t do is setup a cloud provider load balancer for you.

For persistent storage, Nomad uses CSI which is the same technology K8s does: https://learn.hashicorp.com/nomad/stateful-workloads/csi-vol...

Logging should be very similar to K8S. Both Nomad and K8S log to a file and a logging agent tails and ships the logs.

Disclosure, I am a HashiCorp employee.

p_l5y ago

Thanks, definitely widened my understanding of Nomad in pretty short time :)

Kinda feels bad that I don't have anything to use it on right now.

VectorLock5y ago

Does the Nomad WebUI support any kind of auth or just the Nomad-Bearer thing?

Thinking about completing my Hashicorp Bingo card.

1 more reply

tetha5y ago

Those are the advantage and the problem of nomad. We're using it a lot by now.

Nomad, or rather, a Nomad/Consul/Vault stack doesn't have these things included. You need to go and pick a consul-aware loadbalancer like traefik, figure out a CSI volume provider or a consul-aware database clustering like postgres with patroni, think about logging sidecars or logging instances on container hosts. Lots of fiddly, fiddly things to figure out from an operative perspective until you have a platform your development can just use. Certainly less of an out-of-the-box experience than K8.

However, I would like to mention that K8 can be an evil half-truth. "Just self-hosting a K8 cluster" basically means doing all of the shit above, except its "just self-hosting k8". Nomad allows you to delay certain choices and implementations, or glue together existing infrastructure.

K8 requires you do redo everything, pretty much.

p_l5y ago

I count the "glue it with existing infrastructure" to be higher cost than doing it from scratch. It was one feature that I definitely knew regarding Nomad, as one or two people who used it did chime in years ago in discussion, but for various reasons that might not be applicable to everyone I consider those unnecessary complication :)

1 more reply

dhash5y ago

- Yes (Traefik, Fabio, consul connect/envoy)

- Yes, just added CSI plugin support. Previously had ephemeral_disk and host_volume configuration options, as well as the ability to use docker storage plugins (portworx)

- I haven’t personally played with it, but apparently nomad does export some metrics, and they’re working on making it better

chokolad5y ago

nomad is strictly a job scheduler. If you want networking you add consul to it and they integrate nicely. Logging is handled similarly to Kubernetes. Cool thing about Nomad that it's less prescriptive

j / k navigate · click thread line to collapse

0 comments

silviogutierrez5y ago

I haven't, and since I've sunk the cost into Kubernetes and know it very well now, likely won't end up there.

In retrospect though, maybe it's exactly what I needed. Great suggestion.

itgoon5y ago

I've been using Nomad for my "toy" network, and I like it. It runs many services, and a few periodic jobs. Lightweight, easy to set up, and has enough depth to handle some of the weirder stuff.

q3k5y ago

Nomad, in its free offering, cannot compete with k8s for organization-wide usage:

- no RBAC

- no quotas

- no preemption

- no namespacing

This means: everyone is root on the cluster, including any CI/CD system that wants to test/update code. And there's no way to contain runaway processes with quotas/preemption.

p_l5y ago

can it handle networking (including load balancing and reverse proxies with automatic TLS) or virtualized persistent storage? Make it easy to integrate common logging system?

Cause those are the parts that I miss probably the most when dealing with non-k8s deployment, and I haven't had the occasion to use Nomad.

adadgar5y ago

What Nomad doesn’t do is setup a cloud provider load balancer for you.

For persistent storage, Nomad uses CSI which is the same technology K8s does: https://learn.hashicorp.com/nomad/stateful-workloads/csi-vol...

Logging should be very similar to K8S. Both Nomad and K8S log to a file and a logging agent tails and ships the logs.

Disclosure, I am a HashiCorp employee.

p_l5y ago

Thanks, definitely widened my understanding of Nomad in pretty short time :)

Kinda feels bad that I don't have anything to use it on right now.

VectorLock5y ago

Does the Nomad WebUI support any kind of auth or just the Nomad-Bearer thing?

Thinking about completing my Hashicorp Bingo card.

1 more reply

tetha5y ago

Those are the advantage and the problem of nomad. We're using it a lot by now.

K8 requires you do redo everything, pretty much.

p_l5y ago

1 more reply

dhash5y ago

- Yes (Traefik, Fabio, consul connect/envoy)

- Yes, just added CSI plugin support. Previously had ephemeral_disk and host_volume configuration options, as well as the ability to use docker storage plugins (portworx)

- I haven’t personally played with it, but apparently nomad does export some metrics, and they’re working on making it better

chokolad5y ago

j / k navigate · click thread line to collapse