Causal Encoder-Decoder

A causal encoder-decoder is a two-stage neural architecture that pairs a large causal (autoregressive) language model acting as encoder with a small, non-causal decoder. The encoder reads the input sequentially and produces contextualized representations; the much smaller decoder consumes those representations in parallel and generates the actual output — hence "asymmetric": encoder and decoder differ sharply in size and mode of computation. The design separates the two most expensive processes: language understanding is performed once by the pretrained causal encoder, while language generation is handled by the fast, non-causal decoder. Because the decoder is a fraction of the encoder's size, tokens are produced with low latency and a small memory footprint — in recent model releases compressed to roughly 890 bytes of KV cache per token. Unlike the classic encoder-decoder transformer (two parts of similar size), the causal encoder-decoder reuses the pretraining of the causal base model unchanged. A pure decoder-only model, by contrast, uses the same causal path for both input processing and output generation, so its working memory grows with every additional token. In practice: causal encoder-decoders fit agentic workflows with long contexts and frequent tool calls, because the encoder state can be cached while the decoder generates quickly from it. The architecture appears in current open-weight models that combine large context windows and low latency on moderate hardware.