ModeSwitch-LLM: A Lightweight Phase-Aware Controller for Cross-Mode LLM Inference on a Single GPU

Large language model inference workloads vary substantially across request types: short interactive prompts are latency-sensitive, long generations are decode-heavy, repeated-prefix chats can benefit from cache reuse, and long-context requests stress memory and prefill computation. We present ModeSwitch-LLM, a lightweight request-boundary controller that improves single-GPU LLM serving efficiency by selecting among fixed inference modes before generation begins. Instead of retraining the model or changing its architecture, the controller uses cheap workload-level features such as prompt length, expected output length, shared-prefix status, memory pressure, batch pressure, and benchmark family to route each request to modes such as GPTQ 4-bit quantization, INT8 quantization, speculative decoding, GPTQ plus prefix caching, and INT8 plus continuous batching. We evaluate ModeSwitch-LLM using Meta-Llama-3.1-8B-Instruct served through vLLM on a single NVIDIA A100 GPU. On deployment-style synthetic workloads, the online controller achieves a 2.10× mean latency speedup over FP16 and a 0.48× mean energy ratio, corresponding to 51.7% lower energy per token. On automatic benchmark workloads used as a quality gate, it achieves a 1.30× latency speedup and a 0.71× energy ratio while maintaining accuracy close to FP16, with a mean accuracy delta of +0.17 percentage points. We also compare against learned routers trained to imitate a constraint-aware oracle, finding that simple rule-based routing remains the strongest practical policy because it adds negligible CPU overhead and avoids more constraint-unsafe choices. These results suggest that request-aware mode selection is a simple and practical path toward more efficient LLM reasoning and serving on constrained single-GPU deployments.