undefined | Better HN

0 pointslobster_johnson9y ago0 comments

One thing I would say is that — because of the aforementioned documentation mess — it seems more daunting than it actually is. And the documentation does make it seem like a lot of work.

All you need to do, in broad strokes, is:

* Set up a VPC. Defaults work.

* Create an AWS instance. Make sure it has a dedicated IAM role that has a policy like this [1], so that it can do things like create ELBs.

* Install Kubernetes from binary packages. I've been using Kismatic's Debian/Ubuntu packages [2], which are nice.

* Install Docker >= 1.9 < 1.10 (apparently).

* Install etcd.

* Make sure your AWS instance has a sane MTU ("sudo ifconfig eth0 mtu 1500"). AWS uses jumbo frames by default [3], which I found does not work with Docker Hub (even though it's also on AWS).

* Edit /etc/default/docker to disable its iptables magic and use the Kubernetes bridge, which Kubelet will eventually create for you on startup:

   DOCKER_OPTS="--iptables=false --ip-masq=false --bridge=cbr0"

* Decide which CIDR ranges to use for pods and services. You can carve a /24 from your VPC subnet for each. They have to be non-overlapping ranges.

* Edit the /etc/default/kube* configs to set DAEMON_ARGS in each. Read the help page for each daemon to see what flags they take. Most have sane defaults or are ignorable, but you'll need some specific ones [4].

* Start etcd, Docker and all the Kubernetes daemons.

* Verify it's working with something like: kubectl run test --image=dockercloud/hello-world

Unless I'm forgetting something, that's basically it for one master node. For multiple nodes, you'll have to run Kubelet on each. You can run as many masters (kube-apiserver) as you want, and they'll use etcd leases to ensure that only one is active.

[1] https://gist.github.com/atombender/3f9ba857590ea98d18163e983...

[2] http://repos.kismatic.com/debian/

[3] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_m...

[4] https://gist.github.com/atombender/e72c2acc2d30b0965543273a2...

0 comments

chrissnell9y ago

You're making things really hard on yourself. Boot your nodes with CoreOS and it provides almost everything you need (except Kubernetes itself) out-of-the-box. It all works really well together and you get automatic updates, too. I can't imagine trying to run the cluster we run on Ubuntu, trying to roll my own Docker/etc/flannel installs.

lobster_johnsonOP9y ago

I'm sure CoreOS is nice, but we're currently on Ubuntu, and I'm trying to reduce the number of unknown factors and new technologies that we're bringing into the mix. Ubuntu is not the issue here. (FWIW, you don't need Flannel on AWS.)

piva009y ago

Can you expand a bit on why you don't need flannel on AWS? We're currently deploying a k8s cluster and I surely went the flannel route (following the steps of CoreOS guide to k8s) but it'd be nice to remove that setup from our deployment if possible.

2 more replies

Rapzid9y ago

Awesome, I'll take a look into all that! This doesn't look too bad. Do you know if you can combine the masters/minions? Our environments are VPC isolated, and we support ad-hoc creation so I'd like to keep server count requirements to a bare minimum.. The current from-scratch guide says it is not necessary to make the distinction between master nodes and normal nodes; and the api, controller, etc appear to be hosted as pods. This makes me happy and makes sense, but then you have something like this which has me confused: https://github.com/kubernetes/kubernetes/issues/23174 .

On a side note, it's pretty awesome how Docker embedded the key-value store into the main binary. Appears to reduce complexity quite a bit.

lobster_johnsonOP9y ago

You can run them on the same box just fine. There's nothing magical about any of those processes.

However, using dedicated masters (by which I mean mostly kube-apiserver) separate from worker nodes is a good idea to avoid high load impacting the API access.

(Just keep in mind that the Kismatic packages I referred to won't support this — you can't install kubernetes-master and kubernetes-node at the same time. But as you discovered, you can run everything except kubelet as pods. On the other hand, kube-apiserver needs a whole bunch of mounts as well as host networking, so to me it seems like you don't gain all that much.)

What is this Docker key-value store you mention?

Rapzid9y ago

https://blog.docker.com/2016/06/docker-1-12-built-in-orchest...

They are using a Raft based store inside the engine now so there is no external etcd dependency. IIRC they are using etcd's raft implementation.

1 more reply

lobster_johnsonOP9y ago

Replying to myself: On Ubuntu Xenial you have to start Docker with this additional flag:

    --exec-opt native.cgroupdriver=cgroupfs

Since Xenial uses systemd, there's no longer an /etc/default/docker. Instead, create /etc/systemd/system/docker.service.d/docker.conf with:

      [Service]
      ExecStart=
      ExecStart=/usr/bin/docker daemon --exec-opt native.cgroupdriver=cgroupfs --iptables=false --bridge=cbr0 --ip-masq=false

TheIronYuppie9y ago

Quick question - if you're using AWS (or GCP or Azure), was there a reason that:

  ./kube-up.sh

Didn't work for you?

Disclosure: I work at Google on Kubernetes

lobster_johnsonOP9y ago

Did you read my earlier comment (https://news.ycombinator.com/item?id=12024148)?

In short, I want and need to understand how it's put together so that I can use it.

There was someone on the #kubernetes-novices slack today [1] who rightly pointed out who described his approach as: Run kube-up, then try to deconstruct everything that kube-up did into a repeatable recipe. I went the other route, by trying to understand what kube-up did and replicating it. I'm still working through things I missed or did wrong.

To be honest, I think Google's approach here is wrong. Kubernetes is being developed at a frenetic pace, but documentation is not being maintained (it's pretty lacking even if you're on GCP!), and users are understandably frustrated with the obscurity of the whole thing. It works, but it takes weeks to gather enough of an understanding of the system, and that's entirely due to lack of documentation.

The documentaton is lacking both a high and low level. At no point does the documentation offer a big-picture view of how everything works together, nor does it offer low-level descriptions of the stack.

I also think the strong focus on kube-up is a mistake, given the lack of docs. I'm sure it works great, but it's not an option for production use, in my opinion. Terraform would have been better here. You're also using Salt — honestly, it would have been so much cooler if kube-up could just take a few inputs ("what cloud?", "what are your credentials?" etc.) and generate a finished Salt config for you, with a separate salt-cloud orchestration config for the provisioning. The current Salt config is a bit of a mess, and not really something you can build on.

Feel free to reach out to me (@atombender) on the Kubernetes Slack if you want to chat.

[1] https://kubernetes.slack.com/archives/kubernetes-novice/p146...

TheIronYuppie9y ago

Great feedback! Both Kubernetes Anywhere (https://github.com/kubernetes/kubernetes-anywhere) and our documentation efforts (https://github.com/kubernetes/kubernetes.github.io) are very much in flight - and they're both coming by 1.4 (~90 days).

j / k navigate · click thread line to collapse

0 pointslobster_johnson9y ago0 comments

One thing I would say is that — because of the aforementioned documentation mess — it seems more daunting than it actually is. And the documentation does make it seem like a lot of work.

All you need to do, in broad strokes, is:

* Set up a VPC. Defaults work.

* Create an AWS instance. Make sure it has a dedicated IAM role that has a policy like this [1], so that it can do things like create ELBs.

* Install Kubernetes from binary packages. I've been using Kismatic's Debian/Ubuntu packages [2], which are nice.

* Install Docker >= 1.9 < 1.10 (apparently).

* Install etcd.

* Make sure your AWS instance has a sane MTU ("sudo ifconfig eth0 mtu 1500"). AWS uses jumbo frames by default [3], which I found does not work with Docker Hub (even though it's also on AWS).

* Edit /etc/default/docker to disable its iptables magic and use the Kubernetes bridge, which Kubelet will eventually create for you on startup:

   DOCKER_OPTS="--iptables=false --ip-masq=false --bridge=cbr0"

* Decide which CIDR ranges to use for pods and services. You can carve a /24 from your VPC subnet for each. They have to be non-overlapping ranges.

* Start etcd, Docker and all the Kubernetes daemons.

* Verify it's working with something like: kubectl run test --image=dockercloud/hello-world

[1] https://gist.github.com/atombender/3f9ba857590ea98d18163e983...

[2] http://repos.kismatic.com/debian/

[3] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_m...

[4] https://gist.github.com/atombender/e72c2acc2d30b0965543273a2...

0 comments

chrissnell9y ago

lobster_johnsonOP9y ago

piva009y ago

2 more replies

Rapzid9y ago

On a side note, it's pretty awesome how Docker embedded the key-value store into the main binary. Appears to reduce complexity quite a bit.

lobster_johnsonOP9y ago

You can run them on the same box just fine. There's nothing magical about any of those processes.

However, using dedicated masters (by which I mean mostly kube-apiserver) separate from worker nodes is a good idea to avoid high load impacting the API access.

What is this Docker key-value store you mention?

Rapzid9y ago

https://blog.docker.com/2016/06/docker-1-12-built-in-orchest...

They are using a Raft based store inside the engine now so there is no external etcd dependency. IIRC they are using etcd's raft implementation.

1 more reply

lobster_johnsonOP9y ago

Replying to myself: On Ubuntu Xenial you have to start Docker with this additional flag:

    --exec-opt native.cgroupdriver=cgroupfs

Since Xenial uses systemd, there's no longer an /etc/default/docker. Instead, create /etc/systemd/system/docker.service.d/docker.conf with:

      [Service]
      ExecStart=
      ExecStart=/usr/bin/docker daemon --exec-opt native.cgroupdriver=cgroupfs --iptables=false --bridge=cbr0 --ip-masq=false

TheIronYuppie9y ago

Quick question - if you're using AWS (or GCP or Azure), was there a reason that:

  ./kube-up.sh

Didn't work for you?

Disclosure: I work at Google on Kubernetes

lobster_johnsonOP9y ago

Did you read my earlier comment (https://news.ycombinator.com/item?id=12024148)?

In short, I want and need to understand how it's put together so that I can use it.

Feel free to reach out to me (@atombender) on the Kubernetes Slack if you want to chat.

[1] https://kubernetes.slack.com/archives/kubernetes-novice/p146...

TheIronYuppie9y ago

j / k navigate · click thread line to collapse