Anthropic recently published a post on how their internal data team enables self-service analytics with Claude. It’s worth reading in full. There are negative results in it, the wins have token and latency costs attached, and at no point does it claim the model does the hard part on its own.
I read it closely because the system it describes is, layer for layer, the architecture we’ve spent the past year building as a product. To make Claude answer business questions accurately, Anthropic needs something between the agent and the warehouse: a governed semantic layer, curated context, and an eval loop. They got one the same way some teams do today, by steering Claude through building it manually: prompting, reviewing, and correcting inside a single repository, with one of the strongest data teams in the industry doing the steering. Most organizations can’t staff that. So the useful question for everyone else isn’t whether the approach works. It clearly does. The useful questions are where the accuracy came from, how much of the manual steering can be automated, and what can be built on top.
The result that matters
The headline number is the jump from 21% accuracy to consistently above 95%. More interestingly, at one point they gave the agent raw grep access to thousands of historical queries, covering every dashboard, transformation, and analyst notebook they had, and accuracy moved “by less than a point in either direction.”
That result rules out a lot. The bottleneck wasn’t model reasoning capacity. It wasn’t a shortage of examples. Adding data access did approximately nothing. What took accuracy from 21% to 95% was structure: a governed semantic layer the agent is required to use, curated reference docs that encode grain, hygiene filters, and known wrong-answer modes, and a validation loop that catches regressions in CI.
We wrote back in March that context-light tools, the ones that read your schema and prompt over table names, demo well and fail on production warehouses. Anthropic ran that experiment at scale on their own warehouse and published the numbers. They have now confirmed: the model is not the most important part of the agentic analytics stack.
The same architecture, named differently
Lay the two systems side by side and the correspondence is strong.
The semantic layer is the mandatory default path. Anthropic’s skill forces every question through the semantic layer first, with the same definitions as the BI tool and the joins, grain, and filters baked in. Raw SQL is only permitted after the semantic layer has been shown not to cover the ask. The skill even lists pre-rebutted excuses so the model can’t talk itself into bypassing the governed path. This part is genuinely interesting and points to what used to be culture passed between engineers becoming technology.
Our inquiry agent enforces the same priority. The semantic layer is the default; the system prompt permits raw schema access only when no semantic layer covers the question. We didn’t get there by reading their post. We got there by building the measurement: an eval harness that runs a fixed question set across three Claude models, with and without a semantic layer available, specifically to isolate that variable. The semantic layer is the variable we keep coming back to, and it’s the one they went looking at too.
The ablation in their post (add context, measure whether accuracy moves) is a question I got interested in a while ago. Six months back I built an open-source research project around it for a talk: Semiosis, a framework for measuring the semantic quality of a context system, grounded in semantic information theory. Instead of running the ablation once and reporting a number, it runs the ablation systematically: progressively removing and corrupting pieces of documentation and semantic layer, measuring how accuracy degrades, and locating the critical components and the redundant ones. Semiosis treats that question as a repeatable measurement: how much of your accuracy depends on structure, and which structure. It was a research project rather than a product, but the framing it forced, treating documentation quality as something you can unit-test, is directly informing how we approach evals now.
Reference docs encode business semantics. Their per-domain docs specify what one row represents, the filter every query in the domain must apply, and the failure modes a senior analyst would flag. None of that information exists in the DDL. In our model the same information lives as structured objects: entities with a defined grain, measures and metrics with default filters, dimensions, relationships. Same content, represented as data instead of prose.
Validation gates every change. Anthropic runs a before/after eval on every meaningful skill edit and puts the delta in the PR description. They grade the agent’s query rather than its output number, because the correct number changes as data refreshes while a correct query stays correct. They pin evals to snapshot dates. We run the same machinery internally: an eval harness, ablation configurations, results captured per model and per configuration. A change that sounds helpful will sometimes regress accuracy, and only a gate catches it.
Where the paths diverge
The agreement is the interesting part. The differences are where the product lives, so I want to be precise about them.
They steer the authoring manually. We generate the first draft from the data. Anthropic’s reference docs are produced by senior analysts directing Claude and reviewing its output, roughly 30 curated files per domain. That’s a sound investment when you employ those analysts and it’s their job. For most organizations, building and maintaining a governed semantic layer is the project that never finishes. So we invert the order: connect to the database, sample real values, compute distributions, cardinality, and foreign-key structure, and generate a semantic layer draft for a human to review. A draft you correct is a much cheaper starting point than a blank YAML file. The human stays in the loop, reviewing entities, refining metrics, and correcting the agent’s guesses.
Their drift defense is process. Ours is structural. The hard part of any semantic layer is keeping it true after the schema changes. Anthropic’s solution is genuinely clever: all data code lives in one repo, a CI hook flags any model change that doesn’t also touch a skill file, and a scheduled agent scans Slack for correction language and opens doc-fix PRs. About 90% of their data-model PRs now include a skill change in the same diff. The mechanism depends on colocation: models, semantic layer, and docs in one repository, maintained by the team that owns all three.
That precondition fails at most companies. Data definitions are scattered across tools and nobody owns all of them. So our drift defense doesn’t rely on a reviewer remembering or on a monorepo existing. The semantic layer is generated from the live database, so regeneration checks definitions against the data itself rather than against documentation that may have rotted. Two pieces of this are near-term roadmap, and I’ll be explicit about that: definition locking, so a curator’s corrections survive regeneration untouched, and a per-edit audit log with agent-versus-human provenance, so the question “who defined this, and why” always has an answer. Both are designed and in the current development cycle. The goal is drift as a property the system checks rather than a discipline the team has to sustain.
Their deployment has one trust boundary. Most companies have several. Anthropic’s warehouse, their analytics team, and the model all sit inside the same organization, so “let an agent query the warehouse” raises no data-residency questions. For everyone else it immediately does. AnswerLayer deploys into the customer’s own AWS account (GCP is on the near-term roadmap). Analysis happens through queries against the customer’s database, and the data stays in the account. By default, model inference runs through AWS Bedrock over a VPC endpoint inside that same account, so prompts and query results don’t transit the public internet either, and installs on private DNS get an internal load balancer reachable only from the customer’s network. The same boundary that satisfies governance also satisfies privacy review. For regulated businesses this is the only deployment shape that passes legal, and it reaches companies that would never clear procurement for a hosted enterprise AI tier.
That last point is the first visible instance of a difference that runs deeper than privacy.
The engine isn’t the product
Read the Anthropic post again and notice what it describes: an engine. A semantic layer, a corpus of curated context, an eval loop. For Anthropic the engine is the whole job, because they serve an audience of one, themselves. Their delivery vehicle is a skill an analyst invokes from a terminal inside the company, and for that audience it’s the right vehicle. It’s also where the two systems stop overlapping. AnswerLayer is not a skill you invoke. It’s an application deployed next to the warehouse, and most of what we build only exists on the far side of that distinction.
A skill’s context ends when the session ends. An application that runs continuously next to the warehouse accumulates state, and several capabilities follow from that:
- Embedding. Anthropic’s analytics answer questions for Anthropic employees. AnswerLayer embeds in our customers’ products, so the questions get answered for their users, and the metrics that move are the customer’s retention and revenue rather than an internal productivity stat. That requires a deployed surface with auth and an API. Today that means embeddable dashboards and scoped API keys; a fuller embed SDK is near-term roadmap.
- A managed eval engine. The post describes the data team maintaining eval cases manually, alongside the system. A standing application can hold those cases as managed objects: versioned, owned, run on a schedule, with results stored per model and configuration. We run that pattern internally today; shipping it as a product feature is near-term roadmap. It’s standard data-science methodology, and as far as I can tell no vertical AI product ships it. It’s the same instinct that produced Semiosis: measurement belongs inside the system.
- Warehouse observability. Running continuously next to the warehouse puts AnswerLayer in a position to hold a longitudinal view of query latency and performance that a session-scoped skill cannot collect. Every query adapter already records execution timing; the longitudinal product on top of it is near-term roadmap, and at least one customer is already asking for it. Optimization follows once you can observe.
- Curation as a service. The post’s emphasis on curation validates how we’ve built, and it quietly assumes a data team with capacity to curate. Most companies have no such team, or have one that’s badly stretched. We supply the curation: forward-deployed engineers who build and refine the semantic layer with the customer. Anthropic demonstrated that curation is essential. We don’t assume customers can staff it.
- A substrate for classical ML. AnswerLayer deployments already serve forecasting and clustering models beside the agent. The near-term step is binding them to the semantic layer, so a forecast of revenue uses the same definition of revenue as every dashboard, served as a stable, inspectable output. That pattern absorbs what were separate product categories a few years ago: the feature store and much of the MLOps stack. The semantic layer was the missing shared definition layer underneath both.
None of this is a criticism of the Anthropic post. They built the engine because no product handed it to them. The engine is necessary but not sufficient in my view, and the distance between the two is the product.
The honest version
I want to be careful not to overclaim. Anthropic reported accuracy numbers from a production system serving a real organization at scale. We measure the same effects and run the same kind of ablations, and we’re building toward the same bar, but I’m not going to wave an equivalent number back. Their post is evidence for the thesis rather than a scoreboard to argue with.
What the write-up demonstrates is that accurate self-service analytics is a data-foundations problem. The model improved over the period they describe, and they pruned scaffolding as it did, but the model was never the limiting factor. The limiting factor was structure: governance, definitions, the unglamorous work of deciding what “revenue” actually means.
Anthropic staffed the manual steering required to build that structure. Not many companies can, which is a large part of why AnswerLayer exists. The structure is the same whether a team steers Claude through authoring it or a product generates it for curation, and the same whether you maintain it with CI hooks or regenerate it from the data. They proved the architecture on their own warehouse. We’re making it something you can point at yours.