Entropy Explained, with Sheep (2016) (opens in new tab)

Almost all of them have Python code to illustrate concepts.

-

1. Entropy of a fair coin toss - https://bytepawn.com/what-is-the-entropy-of-a-fair-coin-toss...

2. Cross entropy, joint entropy, conditional entropy and relative entropy - https://bytepawn.com/cross-entropy-joint-entropy-conditional...

3. Entropy in Data Science - https://bytepawn.com/entropy-in-data-science.html

4. Entropy of a [monoatomic] ideal gas with coarse-graining - https://bytepawn.com/entropy-of-an-ideal-gas-with-coarse-gra...

5. All entropy related posts - https://bytepawn.com/tag/entropy.html

I was extra confused when I discovered that a spread out cloud of hydrogen is lower entropy than the same cloud gravitationally bound together in a star. So entropy isn’t just about “spreading out,” either.

I found that Legos provide a really nice example to illustrate entropy, so I’ll share that here.

Consider a big pile of Legos, the detritus of many past projects. Intuitively, a pile of Legos is high entropy because it is disordered—but if we are trying to move beyond order/disorder, we need to relate it to micro states and macro states.

Therefore, a pile of Legos is high entropy because if you randomly swap positions of the pieces it will all be the same macrostate—ie a big pile of Legos. Nevertheless, each of the Lego pieces is still in a very specific position— and if we could clearly snapshot all those positions, that would be the specific microstate. That means that the macrostate of the pile has an astronomical number of possible microstates — there are many ways to reorganize the pieces that still look like a pile.

On the other hand, consider a freshly built Lego Death Star. This is clearly low entropy. But to understand why in terms of microstates, it is because very few Legos can be swapped or moved without it not really being a Death Star anymore. The low entropy is because there are very few microstates (specific Lego positions) that correspond to the given macro state (being a Death Star).

This specific case helped me grok Boltzmann entropy. To extend it, consider a box with a small ice crystal in it: this has many fewer possible microstates than the same box filled with steam. In the steam, molecules can pretty much we swapped and moved anywhere and the macrostate is the same. With the crystal, if you start randomly swapping molecules to different microstates, it stops being an ice crystal quickly. So an ice crystal is low entropy.

Now, the definition of what counts as a macrostate is very important in this… but this comment is long enough and I still haven’t gotten to the gym…

(Not refuting entropy as the order of time at all, just noting a visual example is not great evidence.)

“The most misunderstood concept in physics", by Veritasium (YouTube, 2023) (https://youtu.be/DxL2HoqLbyA?si=5a_4lCnuv85lRb57)

No-one would believe the scientists explaining that although highly improbable, the uncracked egg does make scientific sense.

I really think education is mostly about providing higher-level intuitions - making correct thought habitual and thus easy.

Part of what's so attractive about this particular article is how it would mesh with related fields (chemistry, statistics, politics, evolution, astophysics, climate science, etc)

It isn’t though.

Entropy is a fancy word for potential distribution over negative potential. Negative potential is the “surface area” over which potential may distribute. The “number of possible arrangements” casually fits into this, yet misses some unintuitive possibilities, like the resistive variance or other characteristics not preempted by who ever constructed the intellectual model.

Idealists insist entropy is a scalar state resolve of delta probability in their model. They are self deceived. Entropy is the existential tendency for potential to distribute toward equilibrium.

As long as boffins can throw away results that do not correlate, they can insist it is anything they like.