SCOPE: A Generative Approach for LLM Prompt Compression

A big issue in modern LLM applications is they tend to feed long context to LLM, which results in high inference cost and latency, and may exceeds the context limit. Prompt compression addresses this issue by reducing the length of input context with minimum loss of generation quality, i.e, the goal of prompt compression is to shorten the LLM input while maintaining a high generation quality. To overcome these limitations, we propose SCOPE, a novel generative method for prompt compression. Unlike the existing token removal methods, our method centers at a chunking-and-summarization mechanism. Specifically, SCOPE splits a prompt into semantically coherent chunks and rewrites the chunks to be more concise. Then the chunks are reconstructed into a meaningful prompt. Additionally, we design several optimization techniques for SCOPE, effectively preserving critical information and text coherence in compression, as well as providing finer grind control of the compression ratio. We conduct extensive evaluation on typical LLM applications like question-answering and summarization. The results show SCOPE achieves a significantly better compression quality, and higher stability than the state-of-the-art methods.