CoreSemDB: Benchmarking Hybrid Semantic-Relational Query Processing over Text-Rich Databases

Real-world data workflows require hybrid reasoning that integrates high-precision structured retrieval with deep semantic understanding of unstructured text. Recently, several database vendors, including Google, Snowflake, and Databricks, have incorporated LLM-powered operators into their systems for querying unstructured data, yet benchmarks for evaluating their end-to-end performance under natural language inputs remain limited. We formalize this setting as NL2SemProg (Natural Language to Semantic Program Synthesis), which requires models to synthesize executable hybrid programs that interleave structured relational operators with LLM-powered semantic operators to resolve queries over text-rich databases. In this paper, we introduce CoreSemDB, a comprehensive benchmark built from 27 real-world text-rich databases. CoreSemDB is organized by an application-driven taxonomy spanning structural complexity across single and nested queries, along with core reasoning abilities including semantic inference and multi-table association. Our evaluation of 15 large language models on 330 queries shows that NL2SemProg remains challenging, with Gemini-3-Pro achieving only 23.64\% accuracy. Further analysis reveals significant bottlenecks in nested compositional queries, trade-offs between precision and recall in multi-table contexts, and domain-specific strengths across different models.