Developing with Docker (opens in new tab)

I don't think I agree with this. Docker is an amazing tool, I've used it for everything I've done in the last 7 years, but this is not how I'd approach it.

1. I think the idea of local-equal-to-prod is noble, and getting them as close as possible should be the goal, but is not possible. In the example, they're using a dockerized postgres, prod is probably a managed DB service. They're using docker compose, prod is likely ECS/K8S/DO/some other service that uses the image (with more complicated service definitions). Local is probably some VM linux kernel, prod is some other kernel. Your local dev is using mounted code, prod is probably baked in code. Maybe local is ARM64, and prod is AMD64.

I say this not because I want to take away from the idea of matching dev and prod as much as possible, but to highlight they're inherently going to be very different. So deploying your code with linters, or in debug mode, and getting slower container start times at best, worse production performance at worse - just to pretend envs which are wildly different aren't different seems silly. Moreover if you test in CI, you're much more likely to get to a prod-like infra than a laptop.

2. Cost will also prohibit this. Do you have your APM service running on every dev node, are you paying for that for all the developer machines for no benefit so things are the same. If you're integrating with salesforce, do you pay for a sandbox for every dev so things are the same. Again, keeping things as similar as possible should be a critical goal, but their are cost realities that again make that impossible to be perfect.

3. In my experience if you actually want to achieve this, you need a remote dev setup. Have your code deployed in K8S / ECS / whatever with remote dev tooling in place. That way your DNS discovery is the same, kernels are the same, etc. Sometimes this is worth it, sometimes it isn't.

I don't want to be negative, but if one of my engineers came to me saying they wanted to deploy images built from their machine, with all the dev niceties enabled, to go to prod, rather than proper CI/CD of prod optimized images, I'd have a hard time being sold on that.

This demonstrates the most pernicious thing about Docker: it is now easier than ever for someone to design a Rube Goldberg machine and then neatly sweep it under the rug.

When you see the dev environment setup described, the knee jerk reaction should be to simplify it, not to automate the running of 30 disparate commands. Then you can much more easily run it in production, instead of boxing up the mess and waiting until you actually have to debug it.

Including developer tooling in a Docker image is missing one of the really useful things about Docker: not needing all that stuff. By using a multi-stage build, you can do all the slow dev stuff at image build time, then only include the output in the image, and that includes things like building a library which wants a different set of conditions to build than your application wants to run.

It also adds an additional level of risk - if your image is compromised, but all that is running in it is your app, oh well. If it's compromised, and it's able to call out to other parts of your stack (yes, some of this is down to the specific deployment process), that's much worse.

Docker is a deployment mechanism. This means publishing Docker images is a deployment activity not a development one.

I don't think software developers should publish Docker images at all [1]. This is a huge impedance mismatch with serious security implications. In particular, your Docker image needs a regular release cadence that is different from your software releases.

Including a Dockerfile is fine, they allow the person doing the deployment to customize/rebuild the image as needed (and help with development and testing too).

[1]: Though I'm not saying you can't be both a developer and sysadmin in your organization. Are you?

I love Docker (more so the idea of containers). Use it almost everywhere: self hosting services, at work everything is deployed as a docker container.

Except local development. Absolutely hate the "oh need to add a dependency, gotta rebuild everything" flow.

I do use it if the project I'm developing against needs a DB/redis/etc, but I don't think there's a chance I'm going back to using it for local development.

In fact, at work, the project where we do use docker in development actually causes the most headaches getting up and running.

I use a combination of CPU architectures, so the idea of running _exactly_ what's in production when developing is already out the window.

> What if running the linters was as easy as: $ docker compose exec web /scripts/run-linters

This seems to ignore the fact that I also run linters in my IDE to get immediate feedback as I’m writing code. As far as I know there’s no way to combine these two approaches. Currently I’m just careful to make sure my local ruff version matches the one used in CI.

It may be possible with VS Code dev containers, but last time I looked at those I was turned off by the complexity.

This is all very good and true, but as usual the devil is found in the details. For instance, my company sells Docker images that depend on a very old and recently unmaintained binary. Over the years, I've found issues with that binary that make it very hard to be sure issues are completely reproducible from system to system (or, as the article suggests, from local to production). Sometimes, it's as simple as a newer base image updating a core dependency (e.g. Alpine updating musl), but other times it seems like nothing changes but the host machine, and diagnosing kernel-level issues - say, your local Mac OS' LinuxKit kernel versus your production Amazon Linux or Ubuntu, and don't forget x86 emulation! - make "test what you develop and deploy what you test" occasionally very daunting.

> What if running the linters was as easy as:

> $ docker compose exec web /scripts/run-linters

What do people do for making these kinds of commands less verbose and easy to remember?

We've done things like use Makefile with the above behind `make lint`. However, chaining together shortcuts like "make format lint test" gets slow because "docker compose" for each one takes time to start up.

If you instead run the Makefile while you have a terminal open inside one of the Docker containers, that can be faster as you can skip the "docker compose" step, but then not every Makefile target will be runnable inside a Docker container (like a target to rebuild the Docker image), so you have to awkwardly jump between terminals that are inside/outside the Docker container for different tasks? Any tricks here?

But wait how do you write code if the services are all in docker like that?

What about my strongly typed monorepo?

(I prefer a variation of this where all artefacts like databases are in docker compose, but my monorepo services run outside docker)

I’ve been writing software for 20+ years. Sometimes I feel like I’m the only one who hasn’t had the problems Docker solves.

The one thing I feel like is missing in guides like this is key management. I don't like the idea of putting secret keys in my compose.yaml and I would prefer to use something more... controllable? Auditable? The thing is, I don't really know, because this isn't the kind of stuff I work on for $dayjob. But I can't help but feel like there's something missing with key management, and for a noob like me I don't know how to fit it into the larger puzzle.

I have experimented with a local setup in our team.

With the new Docker compose watch functionality I think it works well.

https://docs.docker.com/compose/how-tos/file-watch/

For me this has negated the need for manual mounting.

I combine the above with dotnet watch —-non-interactive in the dockerfile for dotnet and a simple ng serve in our Angular apps.

If new dependencies are added via npm install you can set it so that Docker watch will auto rebuild your container. So it gets around that issue too.

I have a .bat file in our repo that runs the Docker compose action to start up all the needed services and has some powershell to wait until the main UI service is up. When it’s up it auto opens the web browser.

I have a docker container that uses Dozzle (https://dozzle.dev/) for log monitoring across the various services. It can also stop/restart containers if needed.

I also have a container that can be ran to perform a database restore from an external Postgres DB into a local Postgres Docker container.

I will say that dotnet debugging is clunky. You can attach to the Docker container in Visual Studio but if a hot reload has happened you can’t debug again until the app has restarted. For dotnet if I need to do some intensive debugging I tend to spin it up outside Docker for this reason.

It sounds good, but Docker is also good for all sorts of other patterns too.

For instance: Docker is great for local development that never touches prod.

Docker is great for hybrid local dev, where you run some services in docker but not others.

If your desktop is Linux and you are building python based web services, then running python in a virtualenv is often much more responsive than having to rebuild some docker thing.

    Developer Tooling

    This is where I tend to run into the most pushback on this pattern but it's also the 
    part that can greatly reduce headaches. Are you ready? Your immutable image includes 
    everything you need for development: linters, tests, and debugging modules. I will 
    sometimes even include a few useful system tools like netcat or ping, as well as a 
    fancy prompt.

    None of these things are necessary for production. They are at best, image bloat, 
    adding anywhere from 100 to 200 MB of useless code to your image that's never used in 
    the wild. Why then, would we want to include it?

Sorry, but this is dangerous advice. This won't pass most serious security audits and to use these tools, you'd likely need to be running as root.

Much better is to strip your immutable images to the bare minumum and instantiate a debug sidercar, eg [1], if you need to peer inside.

[1] - https://github.com/mhoyer/docker-swiss-army-knife

I find this topic very interesting. On one hand, I am sure that every person reading this has run into a ‘bug’ introduced because of a dev/prod mismatch. On the other hand, the only way to get a total match is to either pay an obscene amount of money or roll everything yourself (which will also have cost, much of which can never be recovered).

As an example, I can build something, deploy it on my own and create a near one to one match. But that means building everything and never using a managed service. If the application interfaces with another tool like Salesforce, do we have multiple instances for every single developer?

Or, do we roll our own CRM?

Matching is great but in a managed world it’s very expensive.

It seems I do the opposite of many commenters.

I do not run Docker in production at all but I also do not develop any serious projects outside of Docker.

Installed on the host machine are only VSCode, Docker and Git.

You work on a project by cloning it, opening in VSCode and clicking on "Reopen in container".

This will spin up generic services like databases and then the actual app container as a VSCode Remote Container, with all the development tooling inside the container.

Does not matter if tooling changes between projects, any project can be worked on with a single click of "Reopen in container".

Host machine stays clean.

At work, we took the radically different approach of not having such a thing as a local environment. It won't necessarily work for every tech stack, but we mostly use lambdas, RDS, SQS, dynamoDB, kafka, and S3, so it's trivial to spin up and tear down the stack as we go. Essentially, instead of trying to ship the local machine to prod, we bring prod to the local machine.

It's a breath of fresh air not having to maintain a separate local environment.

As someone new to Docker, the thing that is never answered (including in this post, the irony of the title...) is what an actual dev workflow looks like.

Let's say I'm working on a web app. I start my container, awesome. Now I make a code change. What do I do? Since the code is copied in the container, do I have to stop, rebuild the image, and start again? Or does it automagically rebuild like most frameworks support? And how about debugging, are there any hoops to jump through connecting a debugger to a container? And what about the filesystem. If I need to inspect the output of something, say a log file in the container, is that easy to do? I've tried this before and got pretty lost trying to access the filesystem.

None of these questions are obvious to Docker beginners like myself. We get the whole "consistent environment" benefits of Docker, now talk about a practical workflow please. :)

edit: thanks for the answers!

> At it's simplest, stuff like this: if ENVIRONMENT == "prod": do_something_only_production_does() ...shouldn't happen.

This is a common refrain among people that IMO do not have a lot of experience in big, complex systems, especially ones running a lot of legacy code. Like, ideally, sure, but in reality making this possible almost always involves extra cost, time, and complexity, and what do you gain from that, really? It's a pretty concept, but not at all practical.

That last point about differences between dev, test and prod should be right at the top. It's rare to find teams that have set themselves up for success (for reasons I do not fully understand)

>but importantly, the image at each of these stages is exactly the same: reproducible at every turn

This is wrong. Hadn't read any further.

Regards,

NixOS adept

I was hoping the article was about using a Docker container *as* your development environment. Talk about easily compartmentalizing and standardizing your projects across machines.

Imagine if you got to a new job and your setup instructions were literally `docker pull mycorp:development.mobile && docker run mycorp:development.mobile`.

- Oh, your machine was lost? Take your new machine and docker pull docker run.

- Someone migrated project A to python 3.8 but project B still requires python 3.6? docker pull docker run.

- You have to work on some backend code? docker pull docker run.

- But the backend code uses an IDE that I don't have installed. Docker Pull! Docker Run!

Every job I've ever started has come with a sheet explaining how to set your machine up to work on the codebase. Install this, upgrade that, change this environment variable, and for the love of god don't install a newer version than this. I mean, I know I've been lucky to get a sheet, but this is usually the first full day of work.

I've never really tried to get this going. I assume that it'd be possible to get an OS instance up in docker that you just VNC into. Load up that image with the tools you need (IDE, git, etc.), wire up some stuff for security (GitHub keys, probably other stuff), and clone the initial commit of your repo so that people can just `git pull` the rest.

Maybe this is a thing. I haven't looked in a few years.

Thank you. This is a great resource!