H$^{2}$MT: Semantic Hierarchy-Aware Context-Time Memory for Beyond-Window Language Modeling

Long-context language modeling requires mechanisms for organizing and selectively accessing information that lies beyond the context window. Existing compression and retrieval methods often treat context as a flat token stream: compressed-memory methods usually process linear segments, while retrieval-augmented generation typically appends retrieved raw text to the prompt, increasing prefill cost and weakening control over the context budget. We propose H$^2$MT, a semantic hierarchy-aware context-time memory framework for beyond-window language modeling. H$^2$MT represents a long document as a rooted semantic tree, constructs cached latent memory embeddings for all nodes through bottom-up aggregation, and performs coarse-to-fine query routing at inference time to select a compact set of relevant node memories. This design preserves document structure, supports reusable offline memory construction, and conditions a frozen backbone on selected latent memories rather than concatenating long raw-text contexts. Across LongBench QA benchmarks and structured technical-document QA, H$^2$MT shows favorable quality-efficiency trade-offs relative to memory-token and retrieval baselines: it substantially reduces time-to-first-token and often peak GPU memory compared with memory-token baselines, while exposing a principled trade-off against raw-text retrieval methods, which preserve more lexical evidence but incur higher generation cost. These results suggest that semantic hierarchy can serve as an effective context-time memory structure for practical beyond-window inference.