AI Inference

AI inference is the process by which a trained machine learning model processes new input data to generate predictions, text, images, or other outputs. Unlike training — where a model learns from datasets and adjusts parameters — inference uses a fully trained model to perform specific tasks in real time or batch mode. The economic distinction is fundamental: training a frontier LLM costs $1M–$100M+ as a one-time expense. Inference, by contrast, occurs with every user request — thousands to billions of times daily. As millions of users interact with AI services, cumulative inference costs far exceed training costs over the deployed model's lifetime. Key metrics include Time-to-First-Token (TTFT) measuring latency before the first response token, and Tokens per Second (TPS) measuring throughput. Infrastructure choices divide between batch inference — bulk processing with latency tolerance — and real-time inference requiring sub-second response for interactive applications like chatbots and coding assistants. Optimization techniques span multiple layers: quantization (FP32 → INT8/FP4 for 2–4× speedup), model pruning, speculative decoding, and KV-cache optimization. Specialized inference chips — NVIDIA H100/B200, Google TPUs, Groq LPUs — provide orders-of-magnitude improvements in throughput and energy efficiency. Hardware advances (Hopper → Blackwell → Vera Rubin) drive 2–4× cost reductions per token generation, making previously uneconomical use cases viable.