You have your data in a nice relational structure. Let’s say it’s a product catalog. You ask your retrieval-augmented generation (RAG) system how many products are in your catalog, and it returns an uncountable pile of semantically-related snippets. Request every product in a category, and you get a handful but not all. Ask for everything under fifty dollars with certain features, and matching by meaning seems to be unable to filter by an simple number.

The standard fixes work at the edges. Prompt engineering leaves the core failure intact. Better embeddings raise the ceiling by a few points. Bigger context windows postpone the failure without changing its shape. When your product only has to answer open-ended questions, the pattern is invisible. When it has to answer precise ones, you hit the wall. That wall is structural.

Why RAG fails on structured data: proof from DeepMind

Google DeepMind’s LIMIT paper proves the ceiling directly. For any fixed search index, there are sets of correct answers that meaning-based search can never fully return, no matter how good the embedding model gets. The paper demonstrates it on a toy task: a list of lines like “Geneva Durben likes Quokkas and Apples” with questions like “Who likes X?” On this deliberately simple example, even the best current models return under 20% of the correct answers.

Twenty percent on a toy task where every fact is a single short sentence. On a real catalog with real ambiguity and real overlap, the ceiling is lower. If the ceiling is mathematical, then a better model will not help you. The architecture around the model has to change.

The pattern that works: hybrid retrieval for structured data

The pattern is not new, it just hasn’t spread as quickly as vector RAG has. Take the parts of your data that structural questions depend on and pull them into typed tables the assistant can query. Keep vector search for the parts that stay text. The assistant then answers two ways from one place, a precise query over structured data when the question is structural, a meaning-based search over text when it isn’t.

Three components make it work:

1. A relational store that domain experts can edit directly. This is where the entities and their attributes and the links between them live. Products, categories, prices, relationships. The people who know the domain best have to be the ones editing it, because engineers do not know which product goes in which category or which regulation applies to which contract.
2. A backend the assistant queries. A graph database or a SQL store. This is what the assistant hits when it needs a precise structural answer. The data flows from the relational store into this backend on a schedule or on demand.
3. A vector index for the text that stays text. Manuals, articles, long documents, anything where the value is in the meaning and not in the shape.

The assistant reads a question, decides which retrieval path matches (or uses both), and answers. This is hybrid retrieval, but the specific architectural shape matters more than the label.

What to look for in the tool that holds your structured data

The reason hybrid retrieval isn’t more widespread is that the first component, the relational store domain experts can edit, is harder to source than it looks. Any tool considered for that role needs:

1. Subject experts have to edit the data model themselves, not just the data. When a new question surfaces that needs a new entity or a new attribute, a subject expert should be able to add it without a ticket to engineering. This rules out raw databases and most ORMs, where any schema change is a code change.
2. The interface has to be familiar to non-engineers. Subject experts are not going to learn a new tool for one job. They will use it if it looks like a spreadsheet. They will not use it if it looks like a database admin console. This is where developer-first tools like NocoDB and Baserow fall out. They are excellent for what they are built for. They are not built for the non-technical domain expert.
3. The API has to be first-class. The whole pattern depends on the assistant reading the data and the data model from the store on demand. Tools where the API was added later, where API access is limited by tier, or where the rate limits make production use impractical, do not clear this bar. This is where Airtable and most similar tools fall out.
4. Self-hosting from the start. Clients with data-residency requirements cannot use a cloud-only tool. Neither can teams working with regulated data, or teams that need to keep the assistant running inside their own network.

The tools that clear all four bars are a small set. Grist is one of them, and it is the one Epoch8 chose after evaluating the alternatives directly. It runs self-hosted, the REST API is comprehensive, the interface is a spreadsheet, and both the data and the data model live in tables a subject expert can edit without engineering support.

Why the data model matters more than the RAG tool you pick

Once you have a tool that fits these criteria, the data model is the actual hard part of RAG for structured data.

The first mistake teams make is trying to describe everything at once. The data model should start with three to five entities and three to five attributes each, chosen from the questions the assistant actually has to answer. As new questions surface, new entities or attributes have to earn their place by making at least one question answerable that wasn’t answerable before.

The second mistake is trying to extract the data model automatically. Language models are good at many things, but deciding what the entities in your business are and how they connect is not one of them. That work is domain expertise, and it has to be done by humans who know the domain. Once the model is defined, the assistant can populate it, evaluate against it, and generate answers using it.

The third mistake is treating the data model as static. The structure grows with the questions people ask. Every question that returns a wrong or incomplete answer is a candidate for a new attribute, a new link, or a new entity. Building the model is a loop, not a project.

Empirical proof: from 40 to 60 percent accuracy to 90 percent

Epoch8 has been building on this pattern in production across client work in retail, e-commerce, manufacturing, automotive, and legal. Their AI platform, Vedana, uses Grist as the source of truth for both the data and the data model, Memgraph as the graph backend the assistant queries, and pgvector for the text that stays text.

Measured against a benchmark of test questions, plain vector search returns correct answers on 40 to 60 percent of the set. Vedana answering over the graph returns correct answers on 90 percent or more. That is the accuracy gap the pattern closes. The full architectural walkthrough, the anonymized data model, and the evaluation loop are all in the Epoch8 case study.

Where to start building the structured layer

If you are somewhere in this picture already, the practical starting point is a list. Write down the ten or fifteen questions your assistant has to answer that vector search cannot. Group them by the entity they depend on. Those entities become your first tables. Their attributes and the links between them come next.

Then wire it into your existing RAG stack. The assistant reads a question, decides whether it is structural or open-ended, and answers from the right place. On the structural questions, the accuracy jumps.

Live structure demo in Grist

This document was created by Olga Tatarinova as part of the webinar demonstrating how her team uses Grist. This specific example is designed for a legal context, but the workflow suits any application that requires precise information retrieval. This document itself represents the structured metadata modified by Olga’s team and referenced by AI agents to write accurate SQL against a client database. Open the full template to see the entire structure.

Common questions

How is RAG for structured data different from Text2SQL?
Text2SQL agents translate a natural-language question directly into SQL against an existing database schema. The pattern described here is different in two ways. First, the schema is not just any database schema. It is designed for the assistant to reason over, with plain-language descriptions of every entity and attribute so the model knows what to use when. Second, the pattern combines structural retrieval with vector search on text, so the same assistant handles both structural and open-ended questions from a single interface. Text2SQL is a component that can plug into this pattern. It is not the whole pattern.

Does this pattern work with local LLMs?
Yes. The pattern is model-agnostic. Any model that can read a schema description and produce a query language output (SQL, Cypher, or a structured search request) can use it. Local models like Llama, Mistral, and Qwen work, though smaller models generally need clearer schema descriptions and simpler query languages to hit the same accuracy. The Epoch8 team primarily uses Claude for production work, but their platform supports swapping in other models.

In our MCP demo, Stan uses smaller models like Claude’s Haiku and OpenAI’s Codex Mini to great effect, since the relational structure is so effective.

How do you decide what to structure in a RAG system?
Look at the questions the assistant has to answer, not at the source material. Any question that needs counting, filtering, aggregation, or a precise link between entities is a signal that the underlying data should be structured. Questions that ask for explanations, summaries, or “what does this mean” are usually fine to leave as text. Most production systems end up with roughly 20 to 40 percent of their content structured and the rest handled by vector search.

What is the maintenance cost of a structured RAG system in production?
Lower than most teams expect, once the data model is stable. The maintenance load falls on the subject experts who own the data, not on engineering. New entities or attributes get added by the people who need them, in a spreadsheet-like interface, without engineering tickets. The engineering-owned pieces (the pipeline that flows the data into the backend, the evaluation loop, the query router) are stable enough that most teams touch them once a quarter. The Epoch8 platform template that runs this pattern has been in production for years with the same core shape.

How does this compare to a knowledge graph in Neo4j?
The graph backend in this pattern often is Neo4j or Memgraph or a similar tool. The difference is where the graph is authored. In a traditional knowledge graph deployment, engineers build and maintain the graph directly in the graph tool. In this pattern, the graph is generated from a spreadsheet-like source of truth that domain experts edit. The graph is the derived artifact, not the primary one. That distinction is what lets non-technical people maintain the system.

Learn more about the Epoch8 build

The Epoch8 case study is the closest thing to a full worked example that exists in public. It includes the anonymized data model, the evaluation loop, and the empirical numbers. If you want the visual walkthrough with Olga Tatarinova, the co-founder who built the pattern, the webinar replay covers the same ground live.

If you want to talk through how this pattern would work for your team’s situation, Diam from our team runs 15-minute calls to understand the work and tell you honestly whether Grist is the right fit for the first component.