TEAS: TOKEN- AND ENERGY-AWARE AUTOSCALING FOR COST-EFFICIENT LLM SERVING

Abstract

Production platforms that serve large language models (LLMs) must continuously decide how many GPU replicas to keep online. Conventional autoscalers inherited from microservice stacks scale on request rate or coarse utilization, but LLM requests differ by up to three orders of magnitude in the number of tokens they process, arrive in extreme bursts, and run on replicas whose cold start takes minutes and whose idle power remains a large fraction of peak. We present TEAS (Token- and Energy-Aware Scaling), a horizontal autoscaler for LLM serving that (i) measures load in phase-weighted effective tokens rather than requests, (ii) forecasts load over the replica cold-start horizon with an empirical-residual safety margin, (iii) adapts its utilization headroom to the measured burstiness of the arrival process, and (iv) applies an energy-aware asymmetric hysteresis that delays scale-in until the projected idle-energy waste exceeds the energy cost of a replica restart. We evaluate TEAS in a trace-driven cluster simulator with a calibrated power and cost model, replaying real production traces from Azure LLM inference services alongside synthetic diurnal and composition-drift workloads derived from the real token distributions. Across all four workloads TEAS is the only dynamic policy that sustains at least 96% SLO attainment; on a bursty production code-assistant trace it improves attainment by 32 percentage points over a Kubernetes-HPA-style baseline and by 11 points over static peak provisioning, and on a 24-hour diurnal workload it meets a 95% SLO target at 11.6% lower monetary cost than peak provisioning. We release the simulator, policies, and experiment scripts.