Planned chapter | Dan Hollick
Embeddings and attention.
Embeddings place meaning into a geometric space; attention uses relationships in that space to decide what context matters now.
The source chapter is still planned. This route preserves the collection and offers an original conceptual preview.
Embeddings place meaning into a geometric space; attention uses relationships in that space to decide what context matters now. The apparent simplicity comes from a set of carefully chosen representations, transformations and physical assumptions working together.
Vector meaning
Related concepts often settle near one another because they are useful in similar contexts.
This is one part of a longer chain: symbol becomes embedding becomes attention scores becomes context vector. The useful abstraction hides the physical work, but the underlying constraints still shape the software built above it.
Dot products
Similarity between query and key vectors becomes a score.
The implementation is full of compromises. Precision, speed, storage and energy rarely improve together, so practical systems choose the errors people are least likely to notice.
Weighted context
Normalised scores mix value vectors into a task-specific representation.
Once this layer is visible, familiar design conventions stop looking arbitrary. They are accumulated responses to the capabilities and limits of the machinery below.
A visual study based on the original chapter. Text is condensed and rewritten.