Planned chapter | Dan Hollick
Neural nets and transformers.
Neural networks learn layered transformations; transformers scale that idea by letting every token selectively exchange information with others.
The source chapter is still planned. This route preserves the collection and offers an original conceptual preview.
Neural networks learn layered transformations; transformers scale that idea by letting every token selectively exchange information with others. The apparent simplicity comes from a set of carefully chosen representations, transformations and physical assumptions working together.
Learned weights
Training adjusts millions or billions of parameters so useful patterns emerge from examples.
This is one part of a longer chain: tokens becomes layers becomes attention becomes prediction. The useful abstraction hides the physical work, but the underlying constraints still shape the software built above it.
Tokens
Text is divided into reusable pieces that the model represents as vectors.
The implementation is full of compromises. Precision, speed, storage and energy rarely improve together, so practical systems choose the errors people are least likely to notice.
Transformers
Attention and feed-forward layers repeatedly refine each token’s representation.
Once this layer is visible, familiar design conventions stop looking arbitrary. They are accumulated responses to the capabilities and limits of the machinery below.
A visual study based on the original chapter. Text is condensed and rewritten.