The Semantic Layer: The Most Overlooked Part of Your AI Stack

When you think about AI, you probably think of the frontier labs, the chips, the energy and water that data centers consume, and the wave of AI startups. Those get the headlines and the capital. The semantic layer gets almost none of the attention, and in my experience it does as much as any of them to decide whether AI succeeds. Most business leaders underappreciate it, and many do not understand it at all, because they assume the technology team will handle it.

A semantic layer records what a business means by its words. It holds the agreed definitions of terms such as customer, active account, net revenue and product, and connects each one to the place in the data where the answer lives, so every report, application and AI system works from the same definitions.

At Morgan Stanley, where I spent more than two decades in a range of leadership roles that applied AI, the factor that most set our AI work apart was our data. Every firm had access to the same models. Our content, and the work we put into organizing it, belonged to us.

The first layer: sourcing and quality

Enterprise AI depends on two layers of data work. The first is sourcing and quality: finding the right data, getting it into a usable format and making sure it is accurate, complete and current, which for documents includes keeping the right version. Firms have struggled with this for decades. AI raises the stakes, because it reads millions of records in seconds and answers with the same confidence whether the data is right or wrong.

The second layer: shared business definitions

The semantic layer sits on top of that data. A useful way to understand it is through four elements: concepts, metrics, relationships and mappings. Concepts are the things the business cares about, such as customer, account and product, each with a precise definition. Metrics are the calculations the business relies on, each with a stated formula. Relationships describe how the concepts connect: a customer holds accounts, an advisor serves clients. Mappings connect those definitions and calculations to the approved data sources, identifying the tables and fields that supply the answer.

A bank shows why this matters. Its customer relationship system counts households, its core banking system counts account holders and its marketing system counts email addresses. Each count is legitimate for its own purpose. The trouble starts when an AI system asked how many customers the bank has uses whichever number it finds first. A semantic layer names each definition and records when each one applies.

Documents need the same treatment, and in my experience this is where a semantic layer helps most. A document's product, jurisdiction, effective date and approval status may appear in its text but are often not recorded in a form an AI system can search on. For documents, the semantic layer takes the form of a taxonomy, an agreed list of topics, and metadata, labels on each document that record those details and link it to the concepts the data uses. A product's disclosure document, for example, carries the same product code as the trading system.

Why AI does better with a semantic layer

Without a semantic layer, an AI system has to guess. It reads the abbreviated names databases give each field, decides which tables to combine and picks its own definition of revenue. With a semantic layer, it translates the question into defined concepts and metrics, and the layer builds the query the same way every time, or reports that it cannot answer.

The same benefit reaches people. For years, the firms I have worked with relied on a few experts who knew the quirks of the data, and nearly every request went through them. A semantic layer records that knowledge, so anyone can get the right data and content without deep expertise in the underlying databases.

This April, dbt Labs, which sells a semantic layer, published a small benchmark in which AI models answered eleven business questions about an insurance company's data, twenty times each. With Claude Sonnet 4.6, queries written against the original tables were correct 64.5 percent of the time. Restructuring the data, joining tables and making their relationships explicit, raised that to 90 percent. The semantic layer took it to 98.2 percent. Ninety percent can sound good enough. It means one answer in ten is wrong, and a wrong revenue figure looks exactly like a right one.

For documents, the gain comes in retrieval, the step where the AI system finds the passages it will use. Standard search ranks passages by how closely their wording matches the question, and in filings, contracts and policy manuals full of similar language, it often picks the wrong one, such as a version that has been replaced. Metadata lets the system narrow the search to the right product, date and status first, and recent studies found that it consistently improves retrieval. When data and documents share the same concepts, AI can answer questions that cross both, such as which clients hold products whose disclosures changed last quarter.

Why multiple AI agents need shared definitions

An AI agent is a system that carries out a task on its own, such as reconciling accounts or preparing a client review, and firms are starting to deploy many agents that hand work to one another. A pricing agent that counts active customers over 90 days and a retention agent that uses twelve months can both be correct. A third agent that combines their output without knowing which definition each used will produce an answer that looks right and is wrong.

With three agents and people checking the work, someone catches the mismatch. With 200 agents passing work among themselves, there is no practical way to trace where the answers diverged. Most firms are building these systems now, before their definitions are written down. In my experience, retrofitting definitions after the agents are running will cost about ten times as much as building them in from the start, the ratio quality managers know as the 1-10-100 rule.

The definitions have to come from the business

Several vendors sell semantic layer software, and industry reference models such as FIBO in financial services and ACORD's Reference Architecture in insurance supply a starting set of definitions. Both save time. Neither knows where each concept lives in your systems, and the reference models will miss concepts unique to your company, such as a proprietary product or a measure your management team created. Mapping your data and filling those gaps is laborious work that depends on people who know your data. Deciding what each term means and when it applies is a business decision, which is why the semantic layer belongs to business leaders as much as to the technology team. Built on poor data, it gives consistent answers that are consistently wrong.

What to do now

The first step is a formal program to build your company's taxonomy, the agreed set of business concepts and definitions at the center of the semantic layer. It needs a single business owner with the authority to settle disputes between departments, specific deliverables and dedicated, ongoing funding. Technology teams can build the semantic layer. Only the business people who know the concepts can fill it in.

Apply the program first to one workflow that matters, such as client onboarding or claims handling:

• Name an owner for each critical definition, who decides what each term means and when each version applies.
• Assign dedicated data modelers to design how the data is organized and map it to the definitions, which is where most of the effort goes.
• Fix the sources behind those terms and label the workflow's documents with the same concepts.
• Measure answer quality before and after, then move to the next workflow.

Every firm can buy the same models. The firms that get reliable answers from them will be the ones that have defined what their data means.

References

• dbt Labs, "Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update," April 7, 2026. https://docs.getdbt.com/blog/semantic-layer-vs-text-to-sql-2026
• Mishra et al., "A Systematic Framework for Enterprise Knowledge Retrieval: Leveraging LLM-Generated Metadata to Enhance RAG Systems," IEEE CAI 2026. https://arxiv.org/abs/2512.05411
• "Utilizing Metadata for Better Retrieval-Augmented Generation," 2026. https://arxiv.org/abs/2601.11863
• George Labovitz and Yu Sang Chang, "Making Quality Work: A Leadership Guide for the Results-Driven Manager" (source of the 1-10-100 rule)