AI Infrastructure
Superposition
- Definition
- Superposition is an architectural principle in neural language models: multiple semantic concepts share a common dimension in latent space, similar to the overlay of quantum states. The reconstruction error per dimension falls only with the inverse of width, i.e. ∼1/Width. Doubling the embedding width halves the error, but compute load and memory demand grow disproportionately. At a width of 768, the error typically sits around 8 %; at 1,536 dimensions it drops to roughly 4 %. Beyond a model-specific threshold, further doubling brings little gain; a smaller model with more test-time compute can match it. Superposition thus differs from a plain embedding: there one axis encodes a single concept, here multiple meanings overlap on the same position.
- Category
- AI Infrastructure
Deep Dive: Superposition
Superposition is an architectural principle in neural language models: multiple semantic concepts share a common dimension in latent space, similar to the overlay of quantum states. The reconstruction error per dimension falls only with the inverse of width, i.e. ∼1/Width. Doubling the embedding width halves the error, but compute load and memory demand grow disproportionately. At a width of 768, the error typically sits around 8 %; at 1,536 dimensions it drops to roughly 4 %. Beyond a model-specific threshold, further doubling brings little gain; a smaller model with more test-time compute can match it. Superposition thus differs from a plain embedding: there one axis encodes a single concept, here multiple meanings overlap on the same position.
Production-Ready Guardrails