undefined | Better HN

0 pointscookiecaper8y ago0 comments

Sure. Here's one example that I've dealt with in the last week.

We have a server that receives the logs from our kubernetes cluster via fluentd and parses/transforms them before shipping them out to a hosted log search backend thingy. This host has 5 Docker containers running fluentd receivers.

This works OK most of the time, but in some cases, particularly cases when the log volume is high and/or when a bug causes excessive writes to stdout/stderr (the container does have the appropriate log driver size setting configured at the Docker level), the container will cease to function. It cannot be accessed or controlled. docker-swarm will try but it cannot manipulate it. You can force kill the container in Docker, but then you can't bring the service/container back up because something doesn't get cleaned up right on Docker's insides. You have to restart the Docker daemon and then restart all of the containers with docker-swarm to get back to a good state. Due to https://github.com/moby/moby/issues/8795 , you also must manually run `conntrack -F` after restarting the Docker daemon (something that took some substantial debug/troubleshooting time to figure out).

We've had this happen on that server 3 times over the last month. That's ONE example. There are many more!

Containers are a VC-fueled fad. There are huge labor/complexity costs associated and relatively small gains. You're entering a sub-world with a bunch of layers to reimplement things for a containerized world, whereas the standard solutions have existed and worked well for many years, and the only reason not to use them is that the container platform doesn't accommodate them.

And what's the benefit? You get to run every application as a 120MB Docker image? You get to pay for space in a Docker Registry? Ostensibly you can fit a lot more applications onto a single machine (and correspondingly cut the ridiculous cloud costs that many companies pay because it's too hard to hire a couple of hardware jockeys or rent a server from a local colo), but you can also do this just fine without Docker.

Google is pushing containers hard because it's part of their strategy to challenge Amazon Cloud, not because it benefits the consumer.

0 comments

eikenberry8y ago

I think you are suffering a bit from the over-engineering of Kubernetes and Docker and throwing the baby out with the bath water. Containers in general are great for simplifying deployment, development and testing. We use docker currently and it works great, but we are using just docker and using it only as a way to simplify the above. We are deploying 1 application to 1 system (EC2 via ASG).

There is also nothing keeping you using Docker for containers. LXC also works great and it has no runtime, so you have none of the stability issues you can get with Docker. Though I must say Docker has improved a lot and I think it will stabilize and _it_ won't be an issue (not as sure about Kubernetes).

cookiecaperOP8y ago

Sure, I agree that both Docker and k8s (at some level, k8s probably had to have a lot of that complexity to interface with Docker) are overengineered, and that there are better containerization processes/runtimes.

But I still don't think containers are what most people want. People need/want ultra-lightweight VMs with atomized state. NixOS looks promising but I haven't used it yet. It seems to give you a way to deterministically reason about your system without just shipping around blobs of the whole userland. You can also approximate this on other systems with a good scripting toolkit like Ansible.

eikenberry8y ago

All I want is a way to encapsulate an application and its dependencies as a single artifact that I can run, test and deploy. Right now containers are the best way to achieve this but I'll probably be happy with any solution to this problem.

NixOS does look interesting and I've considered playing with it for personal projects, but IMO it is still to fringe for use at work where you need both technical usefulness and a general consensus that it is appropriate (i.e. mindshare).

gtirloni8y ago

It seems deploying thousands of ultra-lightweight VMs with atomized state would still require an orchestration layer. I don't follow how that would remove complexity and/or improve stability.

2 more replies

mschuster918y ago

> You get to pay for space in a Docker Registry?

Nonsense. You can run your private Docker registry or if you want to support stuff like authentication and access control use Sonatype Nexus. Both open source.

> but you can also do this just fine without Docker

Not as easily. You'd need to use VMs with all their associated costs (especially if you use VMware) to provide proper isolation, and the hosting department usually will have day-long phone calls with the devs to get the software to reproducibly run, and god forbid there's an upgrade in a OS library. No problem there with Docker, as the environment to the software is complete and consistent (and if done right, immutable).

cookiecaperOP8y ago

We run a "private Docker registry" ... backed by Elastic Container Registry. I'm sure this is the case with the vast majority. You're certainly right that it's possible, but it's about as likely as using cloud platforms in a rational way to start with (elastic load scaling only).

andrewguenther8y ago

So your problem is with Docker, not containers as a concept.

cookiecaperOP8y ago

Yeah, kind of. The complexity of the current iteration is not wholly the fault of Docker, though I'm sure some of the utilities had to increase complexity to work well with Docker (k8s just barely got support for other container platforms). This was a story about an annoyance/bug/issue with Docker, but I have annoyances with other things too.

Some people did not know how to do any server management before kubernetes became a big deal, so they think kubernetes is the only way to do it. For the rest of us, I don't think there's a lot of value brought by this ecosystem.

j / k navigate · click thread line to collapse

0 pointscookiecaper8y ago0 comments

Sure. Here's one example that I've dealt with in the last week.

We've had this happen on that server 3 times over the last month. That's ONE example. There are many more!

Google is pushing containers hard because it's part of their strategy to challenge Amazon Cloud, not because it benefits the consumer.

0 comments

eikenberry8y ago

cookiecaperOP8y ago

eikenberry8y ago

gtirloni8y ago

It seems deploying thousands of ultra-lightweight VMs with atomized state would still require an orchestration layer. I don't follow how that would remove complexity and/or improve stability.

2 more replies

mschuster918y ago

> You get to pay for space in a Docker Registry?

Nonsense. You can run your private Docker registry or if you want to support stuff like authentication and access control use Sonatype Nexus. Both open source.

> but you can also do this just fine without Docker

cookiecaperOP8y ago

andrewguenther8y ago

So your problem is with Docker, not containers as a concept.

cookiecaperOP8y ago

j / k navigate · click thread line to collapse