Your AI Data Analyst Is Hallucinating Your Infrastructure Into Disaster

Frontier LLMs answer fewer than half of real-world database questions correctly — and the multi-step operational queries your data center team actually needs are even worse. A new paper claims to have doubled that success rate, and the method is less “more AI” and more “put a leash on it.”

What happened

Researchers from ETH Zürich-area institutions benchmarked text-to-query agents against 49.9 TB of real supercomputer telemetry and found that standard tool-using agents — including frontier models — succeeded on only 43% of end-to-end operational tasks. The core diagnosis: LLMs don’t just hallucinate field names, they hallucinate relationships between heterogeneous data sources, and no syntactic check catches those errors before they corrupt results silently. Their fix is “symbolic separation”: the agentic workflow reasons freely in natural language, but all actual data access is routed through an ontology-constrained Virtual Knowledge Graph with deterministic pre-execution validation. Every complex query gets translated into one validated graph traversal rather than a chain of LLM-inferred joins. The result they report: task success climbs from 43% to 86%, silent data-integrity errors are eliminated, token cost drops 2.4×, and a smaller on-premise model outperforms a larger cloud one. The hallucination problem isn’t solved by a smarter model — it’s architecturally constrained away.

Cold read

This is a single-domain benchmark on supercomputer telemetry — a highly structured, schema-heavy environment that is unusually amenable to ontology modeling. The jump from 43% to 86% is striking, but the baseline was set against a “rigid workflow” ablation, not against the best available RAG-with-validation systems or purpose-built text-to-SQL stacks; we don’t know how this compares to the current commercial state of the art in, say, Databricks or Snowflake Cortex. The paper’s knowledge graph approach presupposes someone has already built a clean, comprehensive ontology of your data — which is exactly the expensive, expert-hours-intensive work most startups and mid-market companies have never done and won’t fund. “Prevents silent data-integrity errors” is the most important claim in the abstract and also the one with the least detail on how completeness of the ontology is validated. And 49.9 TB of one supercomputer installation is not a golden dataset for generalized enterprise analytics — it’s a controlled research environment.

What it means for you

• Signal maturity: 2/5 — promising architecture, single-domain proof, no production generalization yet
• Who gets hurt: Vendors selling “natural language to your data warehouse” products built on raw LLM tool-calling with no semantic layer — your silent error rate is now a liability, not a feature gap
• What breaks if this is true: The pitch “just connect the LLM to your database” becomes negligent in high-stakes operational contexts; infrastructure observability and AIOps startups need a semantic/ontology layer or they’re shipping a liability
• Why it might not land: Ontology construction is the hidden cost that turns a clean research result into a 12-month professional services engagement; most buyers won’t pay for it until they’ve already shipped a bad-data incident
• Watch for: An enterprise AIOps or data observability vendor (think Dynatrace, Datadog, or a well-funded startup) announcing a “semantic layer” or “knowledge graph grounding” feature in their AI query product within the next two quarters — that’s the signal this architecture is crossing from research to product

Forecast as of 2026-09-16

By Q3 2027, at least two enterprise AIOps or observability vendors will ship a named “semantic layer” or ontology-grounding feature for their LLM query interfaces, directly citing reliability/integrity concerns of the kind this paper quantifies — but adoption in companies with fewer than 500 engineers will remain under 15% due to ontology buildout costs.

Source: Symbolic Separation: Grounding Deep Agents in Knowledge Graphs for Trustworthy Operational Data Analytics — Baibek Davletiyarov, Junaid Ahmed Khan, Andrea Bartolini. https://arxiv.org/abs/2609.17107v1