Model Residency

Model Residency describes how much of an AI model must remain loaded in memory during inference. It matters because the operational cost of a model is not determined only by the parameters used for one token. A mixture-of-experts model may activate a small subset of experts per request while still keeping a much larger set of weights resident on GPUs or servers. Those resident weights shape memory requirements, concurrency limits, deployment topology and the realistic choice between local infrastructure, private cloud and managed inference. The concept separates compute load from memory load. A model can look efficient in benchmark tables because it uses fewer active parameters, yet still require substantial memory because the full expert pool must stay available. Conversely, a larger architecture can be practical if sparse activation and quantization keep residency under control. For engineering teams, Model Residency turns model selection into an infrastructure question: how much memory must be reserved before the first useful token is generated, and what does that imply for scaling, failover and cost?