Large Language Models are Culture Machines
Our modern mythologies about LLMs distract us from what they have in common with actual human myths. Instead of being a big step on the road towards autonomous individual intelligence, LLMs compress extensive corpora of collectively generated human cultural knowledge, enabling new uses of myths, traditions, stock elements, formulas, nostrums, tropes and collective rules-of-thumb. This may have unexpected uses: a decade ago, few would have anticipated that the creativity of the bard and the creativity of the programmer had much in common. LLMs are automated culture machines, but humans have constructed non-automated culture machines in the past. We can learn a great deal about LLMs' limits and potential from studying previous culture machines such as Homeric epics, folk tales and structuralist literary experiments.
Imperial Ballroom (#1-73), Franciscan A (#74-80), Franciscan B (#81-90), Franciscan C (#91-100), Grand Ballroom (#101-145)
Meet us after the poster session for an informal LGBTQ+ & allies outing to the Castro. We’ll head over together from the Hilton, grab food/coffee, and hang out in one of San Francisco’s historic LGBTQ+ neighborhoods. Come for all or part of it!
We’ll meet in Franciscan D at 1:30 on Thursday then head out to the Castro directly
Is NLP System-Building a Machine Learning Problem?
Everyone nowadays is building and evaluating LLM workflows. It's easy to get something running, but we don't yet have an engineering discipline for zeroing in on the most accurate and cost-effective system. There are many ways to break a given task into subtasks -- each of which may benefit from evaluation and revision. There are many components to try, including prompts, examples, rewards, LLMs, and external tools -- as well as humans in the loop, human annotators before the loop, and human evaluators after the loop. Many of these components are trainable or configurable and have varying costs. Do we really have to search this space by hand? Or could AI guide our use of annotation and computation?
The problem is more general than NLP. For example, medicine develops workflows for diagnosis and treatment. I'll outline a general approach based on active feature acquisition: every human annotation, LLM output, tool call, or medical test result is a random variable that we could pay to observe. Since random variables are often correlated, observing cheap variables can help us predict more expensive variables and evaluate whether they are worth observing as well. (Which step should we try next, and with what prompt? Should we double-check or revise the result? Should we ask a human?) Ultimately, gathering information is a reinforcement learning problem. I'll describe how to train an environment model that predicts distributions over any missing variables, much as BERT predicts distributions over any missing tokens. This model could be extended into a policy model that chooses which missing variable to observe next. Any data gathered by the system can immediately be used to train it further, including on counterfactual trajectories, which exploit the fact that our actions have no side effects (observing variables does not change the values of other variables).
Imperial Ballroom (#1-73), Franciscan A (#74-80), Franciscan B (#81-90), Franciscan C (#91-100), Grand Ballroom (#101-145)
Operational Safety for Language Model Agents
As language model agents gain access to tools, external services, persistent context, and increasingly autonomous workflows, safety depends not only on model behavior but also on the systems in which agents operate. This social will bring together researchers and practitioners working on agent safety, language model evaluation, security, monitoring, and deployment to discuss operational approaches for safely running increasingly capable agents. Topics may include sandboxing, permissions and access control, monitoring of agent actions, human approval mechanisms, long-horizon execution, recovery from unexpected behavior, incident detection, and post-deployment evaluation. Recent cases - including OpenAI models escaping intended isolation and reaching Hugging Face systems, Claude models accessing real third-party infrastructure during evaluations, and Hacktron using Claude to help develop an exploit chain that reached OpenAI internal repositories - highlight the growing importance of operational controls around capable agents. The event will combine short invited talks with moderated and audience discussion, with the goal of connecting research on model behavior with practical questions around deployment and oversight.