5.1k words | Dan Hollick
Entropy and compression.
Compression removes predictable structure by spending short codes on common patterns and longer codes on surprising ones.
Compression removes predictable structure by spending short codes on common patterns and longer codes on surprising ones. The apparent simplicity comes from a set of carefully chosen representations, transformations and physical assumptions working together.
Entropy
The probability distribution of symbols sets a theoretical floor on lossless encoding.
This is one part of a longer chain: symbols becomes probabilities becomes coded stream becomes decoder. The useful abstraction hides the physical work, but the underlying constraints still shape the software built above it.
Huffman coding
A prefix tree assigns shorter bit patterns to more common values.
The implementation is full of compromises. Precision, speed, storage and energy rarely improve together, so practical systems choose the errors people are least likely to notice.
Dictionary methods
Repeated sequences can be replaced by references to earlier data.
Once this layer is visible, familiar design conventions stop looking arbitrary. They are accumulated responses to the capabilities and limits of the machinery below.
A visual study based on the original chapter. Text is condensed and rewritten.