Most context layer platforms look promising at the start. The problem is month six, when four hundred small changes have gone in, two teams define churn differently, and nobody can say when the number moved or who moved it.
Choosing the right solution comes down to three questions.
Below are nine specific things to test, the question to ask in the room for each, and what a bad answer sounds like. At the end there is a scorecard you can take into a demo.
These two get used interchangeably. They are not the same thing, and the difference is the whole reason this guide exists.
A semantic layer, including what some vendors call a metrics layer, was built so the same metric means the same thing everywhere it's used: dashboards, reports, spreadsheets, embedded analytics and, lately, AI tools. It defines a finite set of metrics, segments, entities and features for a known set of questions, built from whatever the team already has: BI tools, query history, data dictionaries and documentation. It has done that job well for years.
A context layer is the natural next step for making that infrastructure AI-ready. It has to answer questions nobody anticipated, using the rules and parameters specific to your business. That needs the semantic catalog (metrics, segments, entities and features), and a lot more around it: a table catalog with the joins, an event catalog with what each event actually means, files, skills (proven ways to answer a type of question), and memory that holds the business rules and the knowledge that currently lives in somebody's head.
Here is the honest version: an AI-ready semantic layer is a good start, but only part of the job.
The rest is everything a human analyst would have known without being told. Which internal test accounts to exclude. That the spike in March was a reinstall wave and never counts as real retention. That orders from the wholesale channel do not belong in average order value.
So when a vendor says "we have a context layer," at least three different systems are hiding behind that sentence:
All three demo beautifully. One question, one correct analysis, applause.
One disclosure before we go further. This guide is written by a vendor. We built one of these. Use it as a question list rather than a verdict, and ask every one of these questions of us too.
The argument about whether context matters is over. Everybody agrees, so everybody now says the word, and the word no longer tells you anything.
Give a model a tidy practice database and it writes correct SQL most of the time. Point the same model at a real company warehouse, thousands of columns, a few different flavors of SQL, and a problem starts. On Spider 2.0, a benchmark built from real enterprise databases, the same text-to-SQL method that scored 86.6% on the original Spider scored under 6% on Spider 2.0-Lite when the benchmark launched. Today the top of that leaderboard is above 75%, and the systems near the top are agents built around the model, with retrieval, context and checks, not a model on its own.
The model did not get worse. The data got real.
Real data is complex and messy, and much of what makes sense of it was never written down. What closed most of the gap on the benchmark wasn’t a bigger model on its own. It was the system built around it - and in your warehouse, that system starts with the context layer.
Everything in this part is about what happens when something changes after the context layer is already live.
In reality, things change constantly, and that's healthy. A layer that stops changing is a layer that becomes obsolete.
Three things commonly go wrong when changes go unchecked.
In all three cases the SQL is valid. That is exactly why they survive.
The fix is in the process: enforce an audit on every addition and edit, before it lands, and let it run continuously and on request. The agent checks for near-duplicates, for logic that conflicts with something already there, and for anything that would change the answer to a question people already ask. The audit has to look at meaning, not names, because "net revenue" and "revenue after refunds" are the same metric spelled differently.
Ask the vendor: "Show me what happens when I add a metric that duplicates one you already have."
Good answer: the system stops it before it goes live and tells the user exactly what it conflicts with.
Bad answer: it passes through but gets flagged in a report somebody reviews later.
This goes beyond an audit log. You see the change before it lands, soft launch it, and request review and approval. There's a full history, every line has an author, and a rollback takes one action.
The real test is whether a change can be undone as easily as it was made. Anyone can show you that a definition changed.
Fewer can show you who changed it, why, and what it said before, and then put it back in one step. That is what makes an edit safe to deploy.
Ask the vendor: "Show me who changed this metric last, what it said before, and how I roll it back."
Good answer: the full history on screen, with the author and the previous version, and a one-step rollback.
Bad answer: a log file somebody has to dig through.
When a system can create new definitions, the risk isn’t that it creates a bad one. That will happen. The risk is that a bad one goes company-wide the second it's created.
Good staging works the way code ships: creation, a private soft launch only the requester can query, a draft that another data owner reviews, the audit, an optional evaluation against known questions for high-impact changes, and only then deployment to everyone.
An example. A product manager asks for "session length." The agent builds it as the time between the first and last event in a session. Perfectly reasonable. Except the company already defines a session as ending after 30 minutes of inactivity, and that rule lives in the company's existing data models, not in the request.
With staging, the metric is private to the person who asked for it. They query it, see numbers roughly double what their existing reports show, and ask why. It gets fixed before anybody else sees it.
Without staging, that metric is live, quoted in a deck, and corrected a month later by somebody who noticed it didn't match.
Ask the vendor: "Who can work with a new metric on the day it is created?"
Good answer: only the person who asked for it, until a data owner who is not that person has reviewed it and an audit has run.
Bad answer: everyone.
Rerun last quarter's analysis today. If a definition it used has changed since, the change should be flagged right there in the chat, so you know why the number moved. Dashboards built on that definition should update to the new version too, instead of quietly running on the old logic. And the old version shouldn't disappear: the full version history stays available, so you can always see exactly what the metric used to say.
What you should not get is a different number and no explanation.
The other half of this is retiring metrics safely. When a metric is replaced, the analyses and dashboards built on it shouldn't break. They should move to the new definition, with the change flagged so nobody is surprised. Delete a metric without that and you have quietly broken every piece of work built on it.
Ask the vendor: "Rerun something from before the change. What do I see?"
Good answer: the change is flagged in the chat, naming what changed, dashboards that use it are updated, and the previous versions are one click away in the history.
Bad answer: the new number, silently.
This is the part most buyers never test. It’s where solutions that provide only the context layer, without the agent that runs the queries, fall short - and it’s where the ongoing cost actually sits.
If the platform can't autonomously and proactively improve its context layer, a person needs to do it, and that effort is a line in your budget.
Somebody types "no, exclude the reinstall wave, that isn't real retention." That sentence is a business rule, written by the person who knows it, at the moment it mattered, without being asked.
Most setups throw it away. It lives in one chat session. Four weeks later a colleague asks a version of the same question and gets the uncorrected answer.
It should land somewhere as a draft rule with the evidence already attached: which metric it applies to, who said it, and the query that produced the wrong number. Then it goes through the same gate as anything else. The same goes for a thumbs-down: even without a typed correction, it should trigger a diagnosis of what went wrong and a suggested fix.
Ask the vendor: "I correct an answer. What happens next?"
Good answer: it goes into a review queue as a proposed rule, with the evidence attached. A thumbs-down gets a diagnosis and a suggested fix.
Bad answer: it is kept in the chat history.
Over one week, twelve different people ask some version of revenue per paying user by platform. The layer has revenue per user (ARPU) but not revenue per paying user (ARPPU), and platform isn’t a first-class dimension.
Nobody filed a ticket, because each person got something close enough and moved on. Only something that saw all twelve could spot the pattern: the agent itself.
Same for joins it had to guess because no key was written down, definitions matched by name instead of by rule, and metrics people keep asking for that don't exist. A good platform raises these as proposals, with a draft definition and the SQL, before anybody complains, and ranks them in one place so the most important gaps get fixed first.
Ask the vendor: "Show me something the system proposed on its own last month, the evidence it attached, and what happened to the proposal."
Good answer: specific and slightly boring: a prioritized list of gaps in one place, each with a suggested fix.
Bad answer: a roadmap.
An engineer renames a column tonight. Finance rebuckets a subscription tier. A new product category launches and every breakdown by category is quietly missing it.
The layer has to be built to notice it before a user does, and to keep measuring accuracy automatically as data and definitions change, not just once at launch.
It also has to stay in sync with wherever part of this knowledge already lives, like BI tools, data dictionaries, query history, or transformation tools such as dbt, instead of becoming one more copy that quietly drifts.
Ask the vendor: "An engineer renames a column tonight. When will I find out?"
Good answer: the affected definitions are flagged automatically, an owner gets pinged, and automated evaluation shows whether accuracy moved.
Bad answer: when somebody spots a wrong number.
These questions seem less exciting, but they matter most to whoever owns the budget.
This is the least understood piece in the category, and the one that most separates systems that look identical on a slide. It's worth a few minutes.
Say your layer is in perfect shape. Every definition is correct, audited, versioned. Somebody asks for average revenue per user in the US last week, excluding internal test accounts. Every definition needed is in there. The question is how much the system has to read before it finds them.
Without a harness, the agent is working like someone handed a filing cabinet and told to have a look.
Pull the table list. Four look revenue-related, so read all four schemas. Hunt for how the country is stored, and work out whether the country, geo or market is the live one. Search for anything matching "test accounts," miss the rule because it's filed under "internal traffic exclusion," and find it a few passes later. Load some recent examples to copy the SQL style from. Generate. Hit an error on a column name it read fifty turns ago and has since forgotten. Read it again. Generate again.
Every one of those steps is ‘reading’. You pay for all of it. The answer may well come out right, but it costs many times what it should have, the answer takes longer, and it has the potential to get less reliable the more the window fills with material the question never needed.
The harness, sometimes called the efficiency layer, sits between the agent and the context, and does the finding among other things.
Before the model writes anything, it has already been handed four things: the one table that is the source of truth, not four candidates; the dimensions it can use, with their real column names; the rules that apply, including the test-account exclusion, whatever it's filed under; and the SQL that worked on similar questions.
The model doesn't need to search its way there.
All of that searching and filtering runs as plain code, outside the model. Code is cheap. Model calls are the expensive part, so the less the model has to read, the less each question costs.
This discipline is called context engineering, and Anthropic's guide is a good primer.
The part that gets missed. The harness is also what keeps the records. Which definitions resolved cleanly. Which ones it guessed at. Which joins were ambiguous. Which answers got corrected. Which questions produced nothing usable. A good system also shows this to the user, marking every answer by whether it rested on verified knowledge, unverified knowledge or knowledge that is missing.
None of that is visible from inside the context layer, because all of it happens during the process of the answer. Which means everything in Part B depends on it. Without the harness you have definitions and no idea how they're performing, a catalog with good intentions…
On cost savings. We're not going to publish a percentage, much as we'd like to. It depends on schema size and question complexity, and we have no benchmark we can point you to. The shape of it is simple: the difference between a system that reads the warehouse to find what it needs, and one that is handed it.
Ask the vendor: "What does the model actually see when I ask a question?"
Good answer: a short, specific list, and they can show it to you for a real question.
Bad answer: a long explanation about prompts.
Most of what this guide describes adds to the layer. Very little takes anything away.
Without pruning you end up with 800 metrics, 600 of which are variants of 40. Then three things happen at once. Accuracy drops, because there are more wrong candidates to pick from. Cost rises, because searching a bloated space costs more. And trust goes, because users see three metrics called churn and stop believing any of them.
Long context isn't free accuracy either. Models are least reliable at finding things in the middle of a long input, as Liu et al. documented in TACL. Across 18 models tested by Chroma, performance gets less predictable as the input gets longer, even on deliberately simple tasks.
What good looks like: the system spots two metrics that are nearly the same and offers to merge them, scopes each workspace to the slice that's relevant to it while admins keep the full view, and gives everything a status (draft, published, deprecated, archived) so you can tell what's live.
Ask the vendor: "Show me what a marketing analyst sees, and what an admin sees."
Good answer: the analyst sees only the slice of the layer that's relevant to their workspace, and the admin sees all of it.
Bad answer: everyone sees all 800 metrics.
Criteria 5, 6 and 7 only work if the system answering questions is the same system maintaining the layer.
Not for commercial reasons. It's because the evidence those three run on is created during the answer, and whoever runs the queries is the only one who sees it. A context layer bought separately sits downstream of a system it can't see into. It can hold definitions, version them, and serve them. What it can't do is notice that one of them was corrected, argued with, missing, or guessed at.
So the maintenance work doesn't disappear. It just changes owners.
The honest trade, because this guide is worthless without it. Buying the layer separately means you can switch agents later, and keep the one your team already likes. That's worth real money to some teams, and it may be worth it to you. It also means the health of the layer becomes a person's job rather than something the system does. In practice that person has a roadmap, and this work produces no visible feature. So it slips. Not dramatically. Just steadily.
That's a legitimate choice. It's just better made on purpose than discovered in month six.
No names here, and nothing claimed about any particular product. These are four shapes and where each one runs out. Most real platforms sit somewhere between two of them.
Good at: Governed metrics already exist and finance trusts them. A fast path to consistent numbers on known questions.
Runs out when: A new question needs the joins, business rules and event meanings around those metrics, segments, entities and features, which the layer was never built to hold.
Good at: You can switch agents. No lock-in.
Runs out when: You need criteria 5 to 7, because the records they depend on are produced by an agent the layer doesn't own.
Good at: Speed to a first answer. Demos extremely well, and is genuinely useful on a clean schema.
Runs out when: Two people ask the same question differently, get different numbers, and nothing in the system can tell you which is right or why.
Good at: Fits your stack exactly. Nobody else's roadmap, nobody else's pricing.
Runs out when: The maintenance work needs owning forever, and advanced features require engineering. It's the part teams underestimate by the widest margin, and where in-house builds quietly stall.
Working out which two you're between tells you which of the nine criteria to weigh hardest.
That last one applies to us as much as anyone. If a number has no source you can check, it was chosen, not measured.
Everything in this guide rests on one assumption: your context layer will change constantly, and the system has to handle that safely, without someone babysitting it. That assumption is what we built ClarityQ’s semantic context layer on. The audit, the version history, the staged rollout, the confidence badge on every answer, the Inbox that ranks gaps with suggested fixes, and the feedback loop that turns corrections and thumbs-downs into proposed fixes are as much a description of how ours works as a list of what to look for. Which means you already have what you need to hold us to it.
Ask all nine questions of everyone on your shortlist, and start with us :)
