VLM Token Compression through Task-Adaptive Predictor Selection: A System-aware Co-Design Analysis and Practice

Vision-language models (VLMs) have enabled a broad range of applications, yet their deployment remains costly due to the large number of visual tokens often dominates prefill computation and serving latency. While numerous token compression methods have been proposed to mitigate this overhead, most are evaluated primarily as algorithmic prototypes and do not necessarily translate into practical speedups in production serving systems. In this work, we present a deployment-oriented study of VLM token compression. Moving beyond offline accuracy–compression trade-offs, we study the end-to-end path from algorithm selection and design refinement to system-level integration and evaluation. We compare representative compression strategies in terms of both performance preservation and efficiency gains, and introduce a simple yet effective Task-Adaptive Predictor Selection improvement to achieve a better trade-off. We further integrate the method into the vLLM serving framework and demonstrate practical speedups. Our study provides actionable guidance for developing VLM token compression methods that are both algorithmically effective and deployable in real-world inference systems.