From Demonstrations to Directions: Gradient-Optimized Task Vectors at the Embedding Interface

In-context learning (ICL) lets frozen language and vision-language models infer new tasks from demonstrations, but each query must process the same demonstrations again. This repeated context raises prefill compute, context length, and key-value cache use, especially with images. We ask whether the task information in demonstrations can instead be stored once as a single direction at the input-embedding interface. We propose **G**radient-**O**ptimized **E**mbedding **T**ask **V**ector (**GO-ETV**), a supervised adaptation method that freezes the backbone and learns one $d$-dimensional vector per task. GO-ETV adds this vector to mask-selected existing embedding positions and minimizes first-token cross entropy on the answer. At inference, the learned vector is re-injected at the same mask-defined positions, with no demonstrations, no added tokens, no weight updates, and no layer search. Across both language-only and vision-language backbones, GO-ETV outperforms few-shot ICL and prior task-vector methods on text tasks, cross-modality transfer, open-ended visual question answering (VQA), and fine-grained visual classification. Mechanistic and vocabulary-level probes show that the learned vector approximates the effect of demonstrations, aligns text and image queries by task, and selects the task relation while the current input supplies the content.