undefined | Better HN

0 pointsngrilly5y ago0 comments

Thanks. Seems like it’s worth looking into it then. Still curious about the reasons why it’s more efficient.

0 comments

Nomad was designed for efficiency (linear scalability) and performance from the very beginning. One of major aspects of this is that it is monolithic. The single nomad binary contains:

- The CLI for interacting with Nomad clusters via their HTTP API

- The HTTP API

- The "server" agent which contains the scheduler, state store, and Raft implementation

- The "client" agent which runs workloads and has builtin task drivers for docker, plain processes, and more.

Not only is there no dependency on an external state store like etcd, but there's no pluggable abstraction for state storage. The tradeoff is that all cluster state must fit in memory. Pi clusters should use far less than 100mb of memory while something like C2M (6,100 nodes, 2 million containers) used just under 100gb of memory. Memory use scales linearly with cluster size (nodes+containers).

Since Raft and the scheduler are colocated within a single server agent process, many operations can be performed in the replicated FSM as a simple function call instead of if Raft was an external process requiring networking overhead to interact with.

I'm not very familiar with k8s internals, so I'm afraid I can't offer a very detailed direct comparison.

ngrillyOP5y ago

Thanks for that detailed explanation. The fact that all main functions live in a single process probably helps a lot with CPU and RAM usage, compared to k8s where etcd, API server, scheduler and controller manager live in independent processes, which means serialization/deserialization and less optimization opportunities.

j / k navigate · click thread line to collapse