I love saying this but OTP is a really roughneck standard library. They just added shit to it as they needed it without apparently putting too much consideration into the organization, naming, or conventions.

It makes it very powerful but very disorienting and experience gained with one part of it often does not really prepare you for other parts. Usually each specific tool was created by someone who used it immediately, so it's all reliable in its way. But there is a lot of redundancy and odd gaps.

Elixir's almost extreme attention to naming, organization, and consistent convention is almost as far as you can get from this approach too. It's fun to have them in the same ecosystem and see that there are actually pros and cons to each approach.

Here's a trick to confidence in a BEAM system. If you get good at hot loading, you significantly reduce the cost of deployment, and you don't need as much pre-push confidence. You can do things like "I think this works, and if it crashes, I'll revert or fix forward right away" that just aren't a good fit for a more common deployment pattern where you build the software, then build a container, then start new instances, then move traffic, etc.

Of course, there are some changes that you need confidence in before you push, but for lots of things, a bit crashy as an intermediate step is acceptable.

As for understanding the OTP stuff, I think you have to be willing to look at their code. Most of it fits into the 'as simple as possible' mold, although there's some places where the use case is complex and it shows in the code, or performance needs trumped simplicity.

There's also a lot of implicitness for interaction between processes. That takes a bit of getting used to, but I try to just mentally model each process in isolation: what does it do when it receives a message, does that make sense, does it need to change; and not worry about the sender at that time. Typically, when every process is individually correct, the whole system is correct; of course, if that always worked, distributed systems would be very boring and they're not.

Thank you for this post and I'll add a note for people who are seeing this and are maybe discouraged about learning Erlang/OTP/Elixir.

I generally agree with you that learning Erlang stuff can be daunting.

I will say that many things worth doing are not easy! Erlang and the whole OTP way of thinking is tough to learn in part because it is genuinely different enough from everything else that is out there that one's odds of being familiar with its conceptual underpinnings are low.

If you have trouble learning Erlang (and OTP specifically) it's not because you're dumb, it's because Erlang is different.

Learning Erlang is not like learning any other dynamic language you've learned. Learning Erlang is closer to learning a bespoke operating system designed to build reliable low-latency long-running systems. It's a larger conceptual lift than going from one dynamic OOP language to another dynamic OOP language.

It also took me quite a bit of time to understand OTP. In fact, I had to have a project that actually required what OTP offered to really get it.

Two things that definitely helped me understand were reading the somewhat-dated-but-still-useful material on the topic in Learn You Some Erlang, as well as reading through the "OTP Design Principles" section of the Erlang System Documentation.