Memorilla: Latent Semantic Memory for LLMs

The memory of a large language model (LLM) aids in solving complex tasks by remembering relevant information from the past and accumulating new knowledge in-context. However, a growing corpus of knowledge not only increase computational overheads but also hinders the LLMs ability to retrieve relevant information. Here we present Memorilla, a latent semantic memory for LLMs. Memorilla first embeds the corpus chunks in a reusable query-agnostic fashion. Then, for a given text query, Memorilla processes these embeddings to create a fixed-size query-dependent collection of learnable latent vectors. Finally, these latent vectors are projected into the LLM embedding space as soft “memory” tokens, which are then used by the LLM to complete the task. Unlike existing approaches that either retrieve long pieces of text per query or compress the entire corpus in a query-agnostic fashion, Memorilla offers query-aware retrieval of essential information directly into the LLM embedding space and achieves significant context-length reduction. Across eight supervised benchmarks spanning personalization, long-document, and knowledge-intensive QA, Memorilla consistently outperforms Mem0, MemoRAG, and full-context baselines while using 24×, 152×, and 400× fewer LLM input tokens, respectively. Additionally, we showcase the effectiveness of our approach for agentic RL in TextWorld. Training Memorilla from environment rewards increases pass@3 from 0.250 to 0.375 and terminal completion from 29% to 42% compared with the full-context baseline. Overall, our results position Memorilla as an expressive latent memory system while remaining complementary to text-based retrieval for localized fine-grained details.