Last time I began explaining my “CD collection” analogy for entropy; here I’ll pick up where I left off (and hopefully finish — I seem to be writing a lot of trilogies these days). There’s more to say about macro-states, and I want also to get into the idea of indexing.
I make no claims for creativity with this analogy. It’s just a concrete instance of the mathematical abstraction of a totally sorted list (with no duplicates). A box of numbered index cards would work just as well. There are myriad parallel examples.
One goal here is to link the abstraction with reality.
To summarize and review:
◊ There is a CD collection of at least 5,000 CDs (no duplicates). The collection has a sort order defined on it. The nature of the ordering doesn’t matter, only that some property of the CDs allows unambiguous ordering.
◊ We define the sort degree as the minimum number of moves required to restore the collection to sorted order. When fully sorted, the sort degree is zero. The sort degree is our system’s macro-state.
◊ We define the entropy number Omega as the number of permutations of the collection with the same macro-state (sort degree). For sort degree zero (fully sorted), there is only one permutation, so Omega=1. The permutations are our system’s micro-states; there are N! of them (N factorial).
◊ Per Boltzmann we calculate the entropy of a specific macro-state as:
Where S is the entropy of that macro-state and K is Boltzmann’s constant. (We can ignore the constant and treat the entropy as just the log of Omega.) Since Omega=1 for a perfect sort, the entropy of the sorted collection is log(1)=0.
The general idea is that misplaced CDs increase the entropy of the collection. If a perfect sort is one extreme (zero entropy), the other extreme is a random pile — maximum entropy (which has an Omega of something on the order of log(N!), although it’s a bit more complicated than that).
To connect the analogy with reality, we’ll say the owner has initially expended considerable energy sorting the collection. Its zero-entropy state is the result of spending that energy. We can compare it to freezing something down to absolute zero. That takes energy which raises the entropy of the surrounding environment (often as waste heat).
The owner’s many friends borrow CDs and aren’t always perfect in putting them back. Their errors may be small or large. One might return the CD to the right artist group, but out of order within that group. Another might misfile a Neil Diamond CD under “D” rather than “N” — a large error. Someone else might not even try and put returned CDs at the end.
Over time the collection becomes more and more disordered. We compare this to the second law of thermodynamics. The entropy of a system grows over time if energy is not spent maintaining or reducing it.
The owner can either spend that energy in small chunks keeping the collection sorted at all times (keeping its entropy low) or can allow the disorder to grow and spend a large chunk of energy re-sorting it. A thermodynamic equivalent is keeping ice-cubes in the freezer versus letting them thaw and re-freezing them.
Given the collection has at least 5,000 CDs, even several dozen out of place doesn’t massively raise the entropy. If 42 CDs are misfiled (less than 1%), the number of permutations has almost 160 digits. That may seem a big number, but the total number of permutations for 5,000 CDs is a number with over 16,300 digits — over 100 orders of magnitude larger.
Getting back to the micro- and macro-states, a micro-state is a permutation of the collection. Conceptually, it’s a list (or knowledge) of the positions of each CD. Therefore, a micro-state of the collection fully describes it (in terms of its order).
Imagine each CD has two numbers associated with it: The index of its correct position in the collection, and the index of its current location. If the difference is zero, the CD is in the correct position. If it’s zero for all CDs, the collection is sorted.
We generate these by numbering both the CDs and the locations holding them. The CD’s index number corresponds with whatever property of the CD allows us to sort it, but such properties, names for instance, don’t form a contiguous spectrum of indexes. The CD index numbers do, so the CD with index number 2197 is the 2,197th CD in the collection.
In a sorted collection, the location indexes match the CD indexes. When a CD is misplaced, its index won’t match the location’s. Note an important wrinkle: Think about making room on a shelf to insert a CD. You stick your finger between two and make an opening. As a result, the ones on the shelf move a little. If the shelf was previously full, and you removed one leaving a gap, but you make a new gap to misfile the one you took, the other CDs move to compensate.
This happens with the entire collection when we misfile a CD. Here’s an illustration using a collection of only five items. We’ll represent the sorted collection as a list of the position differences: [0 0 0 0 0]
All five CDs are in the correct location. Imagine we move the last one to the first position. Now the differences are: [-4 +1 +1 +1 +1]
All the CDs moved. On the other hand, if we just swap the last two: [0 0 0 -1 +1]
Only two CDs moved. Note that both examples have sort degree one. A single move restores the order. (Note also that the sum of differences is always zero.)
If the sign of these lists is reversed, we get a list telling us what moves to make to each CD to restore it to order. For instance, the end-to-end swap example becomes [+4 -1 -1 -1 -1] which, applied to each CD at that location, restores the sorted order.
Note that both these numbers are, in some sense, virtual. They need not exist concretely. These are virtual views of the collection dynamics. We label things so we can talk about the system. Nature has no need for our labels.
Bottom line, the micro-state, however implemented, fully describes the system at its fundamental level. Here it’s sort order. In a thermodynamic example, it’s the positions and momentums of molecules or atoms.
The macro-state is an overall property of the system, a single number. In thermodynamics, it’s usually temperature or pressure — some physical property usually associated with energy. In the CD collection it’s “how sorted” is the collection (note that “how disordered” is the inverse of “how sorted”). We quantify it formally with the sort degree number.
Sort degree lets us count the number of micro-states for a given degree of disorder. As noted previously, an accurate sort degree in all cases may be impossible, but its abstraction is important. Nature doesn’t care how hard the math is, she just naturally least actions. Plus, we can calculate it, or approximate it, in enough cases to appreciate the system dynamics.
An important point here is that we must disavow the micro-states. In a physical system, we often can’t know them without extraordinary effort (or sometimes at all.) Indeed, thermodynamics (and entropy!) arose from the need to deal with systems with bajillions of effectively invisible pieces.
So, although it’s true one can just look at the CDs and see “how sorted” they are, one has to disavow this ability. (One reason there are 5,000 is to make it harder to spot a few CDs out of place.)
We could imagine they’re sealed in a jukebox, but that changes the metaphor a bit. Friends wouldn’t borrow and misfile, but the jukebox robot might sometimes make mistakes in putting CDs back after playing. The robot has the same choice of expending small constant energy correcting its mistakes, or waiting for some threshold and spending more energy re-sorting.
Note that in both cases the cost of a disordered system is the need to spend more energy finding a desired CD. The more disordered, the harder the search. A random pile is the hardest. The translation is that low-entropy systems can do more than high-entropy systems. It takes more effort to accomplish the same thing in a high-entropy system.
Given our view is restricted to the macro-state sort degree, the tough question is how we measure that number. What kind of “thermometer” can measure “how sorted” the collection is.
As mentioned last time, that turns out to be NP-hard, so our sort meter is necessarily a little magical. (But it’s the only magic in the analogy, which isn’t bad for physics analogies.) We can’t generally take a given permutation and determine the minimum number of moves to re-order it.
But we can cheat and track the disorder with some kind of index.
The progressive disorder of the collection (because sloppy friends or a broken robot) represents how entropy grows in a physical dynamic system where energy isn’t spent fighting it.
But if we spend energy keeping track of each misfiled CD, we’ll always know how many there are. A simple way is, given a sorted collection, start with an empty “error” list. When a CD is misfiled, add an entry to the list tracking it. If the CD is later put back, erase the list entry. We could also just track the location of all the CDs all the time, although this takes more effort.
As the entropy of the collection grows, we put energy, not into correcting filing errors or re-sorting, but in maintaining the error list. These are effectively the same thing. The combination of the collection+index still has zero entropy, but the description of the collection is now divided between it and the index. We would need to expend energy using the index to re-sort the collection and restore it to zero entropy.
An important distinction: When viewed as its own object, the index necessarily always has zero entropy. It must if it’s to have any value. The energy we spend on it is to maintain that zero-entropy condition.
It’s worth mentioning that an index doesn’t have much analogue in thermodynamic systems. It’s more from the information theory side of things. One real example is the sector index on a disk drive. Another is the card catalog of a library.
This all merely scratches the surface of a vast topic. The key points are that:
- Entropy is a measurement we can make (not a force). More importantly, it’s a number, not a process. (The process is thermodynamics or computation or transmission or some other action.)
- In the abstract, it’s the number of micro-states that result in the same macro-state. Any discussion starts by defining these for the system in question.
- The entropy=disorder notion comes from the abstraction, and is best illustrated with simplified abstract examples such as sorted CD collections.
- A simplified abstraction is only the beginning and is only intended to provide a basic sense of the fundamental nature of entropy.
And I think that’s “nuf sed” for now, folks.
Stay highly ordered, my friends! Go forth and spread beauty and light.