The Overhang and the Butler

Transcript

This is The Judgment Layer, from Advisers Give Back — a nonprofit working to increase access to pro bono financial planning, pairing households who can't afford a financial planner with CFP professionals who volunteer their time. Every episode starts with a real development in AI and ends somewhere useful for the firm you run. Written and voiced with AI; directed by Matt Iverson-Comelo, executive director of Advisers Give Back. Nothing here is investment, legal, or compliance advice.

Two ideas from the frontier of AI this week — one about what makes human expertise irreplaceable, one about who your client's AI agent is actually working for.

Picture Ethan Mollick sitting at his desk sometime in the last few weeks, staring at a problem he's been circling for months. Mollick is a professor at the Wharton School of the University of Pennsylvania. He's probably the most careful and honest observer of how people actually use AI day to day. He's not a booster. He's not a skeptic. He's something rarer: a rigorous empiricist in a field full of storytellers. And the problem he's been circling is this: the models have gotten extraordinarily capable, but most people are not getting extraordinarily better results. There is, he wrote last week in his newsletter One Useful Thing, a kind of overhang — a massive gap between what the tools can do and what the people using them are extracting from them. And the gap, he argues, is not a tool problem. It is a human problem. Specifically, it is a problem of four things that humans have and AI doesn't: deep knowledge, wide knowledge, taste, and agency. That framing sounds simple. What it actually implies is one of the most important — and most misread — ideas about AI in 2026.

Here is what most people get wrong when they read Mollick's argument. They hear "humans still matter" and they file it under "AI won't replace us" and they move on. That is the skimmed version. The actual argument is sharper and stranger than that. Mollick is not saying AI is limited. He's saying the humans who will benefit most from AI are not the ones who know how to prompt well, or who use the most tools, or who have adopted the most workflows. They are the ones who know their domain deeply enough to catch the model when it's wrong, broadly enough to connect the output to something unexpected, have the taste to know when "good enough" is actually not good enough, and have the agency to push past the first answer and demand something better. The people who are not extracting value from AI are often not the people who know nothing about AI. They are the people who know nothing about their subject. Because what AI has done, paradoxically, is make expertise more valuable, not less.

Let that sit for a moment. The conventional story — the one you hear at every conference, from every vendor, in every press release — is that AI democratizes expertise. Anyone can now do what used to require a specialist. That is true, up to a point, and that point turns out to matter enormously. What AI actually does is make the floor higher. It gives a novice a plausible-sounding output where before they'd have nothing. But it does not give them the ability to know when that output is subtly wrong, partially right, or right in form but wrong in context. Mollick's point is that the ceiling for the expert who uses AI has risen faster than the floor has risen for the novice. The gap between what a genuine expert extracts and what a casual user extracts is widening, not narrowing. The people who look at an AI-generated answer and say, "that's close, but here's what it missed, and here is the better question" — those people are compounding their advantage at a rate that is genuinely alarming if you are on the other side of it.

And here is where it connects to something Azeem Azhar, the technologist and writer who publishes the Exponential View newsletter, was exploring in a piece this week called "Safety in Numbness." Azhar's concern is subtler: that the sheer fluency of large language models is training people, slowly and almost imperceptibly, to doubt their own instincts. If you ask a model a question and it answers confidently, and you have a faint sense that the answer is slightly off, the socially and cognitively easy move is to defer. The model sounds authoritative. Your doubt feels vague. And so you go with the model. Azhar argues that a flash of genuine disagreement with an AI — that moment when something in you says "that's not quite right" — is not a sign of resistance to technology. It is a sign that your expertise is working. It is the signal you should be amplifying, not suppressing. He calls disagreeing with a large language model a good sign. Most people treat it as a friction to be overcome.

Put Mollick and Azhar together and you get something that does not appear anywhere in the mainstream AI discourse: the possibility that the organizations which invest most aggressively in AI tools without investing equally in deep human expertise will, over some medium-term horizon, end up worse off than they expect. Not because the tools don't work. Because the tools work best for the people who know enough to use them critically, and those people are becoming rarer in organizations that have decided AI is a substitute for developing that knowledge in the first place.

Now turn the lens on the firm you run, and think about what this means in twelve to eighteen months.

The first-order pressure you've felt is efficiency. How do we use AI to do more with fewer people? How do we cut the time it takes to do a financial plan, generate a client summary, produce a proposal? That pressure is real, and the tools are delivering on it. But Mollick's overhang idea suggests there's a second-order consequence that is arriving right behind it, and it runs in the opposite direction. As AI does more of the production work, the comparative advantage inside your firm shifts entirely toward the people who have the judgment to evaluate that production work. The planner who reviews an AI-generated financial plan and spots the edge case the model didn't account for — the asset structure the model treated as straightforward, the behavioral pattern in the client it didn't know to look for — that planner is not just valuable. That planner is, increasingly, irreplaceable in a way that a pure production role never was.

This has an implication for how you develop advisers that is not obvious until you think it through. If the goal of development used to be getting someone from "doesn't know how to do a financial plan" to "can produce a good financial plan," AI has now compressed a significant part of that journey. The model can produce a plan. What the model cannot do is teach someone to have the taste to know whether it's actually a good plan, in this context, for this client, given what this client hasn't said yet. That is judgment. And judgment is not built by watching the model produce plans. It is built by the hard, repetitive, supervised work of explaining your thinking to someone more experienced, being corrected, and doing it again. Which means the most valuable thing you can do in adviser development right now is not to give your newer advisers better AI tools. It is to give them more structured time with your most experienced advisers — not to learn workflows, but to absorb judgment. Because if Mollick is right, the adviser who enters the profession in 2026 and primarily learns by reviewing AI outputs without that overlay of expert correction may emerge in five years technically competent and critically shallow. And you will not be able to buy your way out of that problem with a better AI subscription.

The second implication is for how you staff what you might call the oversight layer. In twelve to eighteen months, every mid-to-large registered investment adviser will have more AI output flowing through it than any human team can review in detail. The question is not whether to have human review. Compliance will require it. Your professional obligation demands it. The question is who does that reviewing, and what makes someone good at it. The answer, if Mollick and Azhar are right, is that the best AI reviewers in your firm are not the people most fluent with the tools. They are the people who are most fluent with the subject matter — the ones who feel that small friction Azhar describes, who notice when something is slightly off, who have the confidence to push back on the model rather than click through. Right now, most firms are not thinking about oversight as a skill set. They are thinking about it as a compliance checkbox. The firms that figure this out first will have a structural advantage that compounds quietly, year after year, because they will be producing better work and catching more errors, and the gap between them and the firms that trusted the model a little too much will only become visible when something goes wrong.

The one-sentence version: AI raises the floor for everyone and the ceiling for experts — which means the most valuable thing you build right now is not an AI capability, it is a human judgment capability that your AI capability can't replace.

The second idea lives in a different corner of this week's frontier, and at first it looks like a consumer story. But it is not. It is a story about power — specifically about who controls the relationship between a person and the information, recommendations, and decisions that shape their financial life.

Here is the scene. Last week, Meta — the social media and technology company — launched a product called Muse. Muse is described as a personal AI agent, a "digital butler," and it runs inside what Meta calls a persistent Linux virtual machine, meaning it is not just answering questions. It is executing tasks, navigating software, maintaining memory across sessions, and acting on behalf of the user over time. It is packaged, as the writer and technologist Simon Willison noted on his blog this week, in a deliberately cute and accessible way — a friendly mascot, easy to install, seemingly harmless. And it is drawing a striking amount of attention, including from John Gruber, the technology writer and publisher of the blog Daring Fireball, whom Willison quoted directly. Gruber's observation was blunt. Muse, he wrote, is "the first consumer-accessible agentic AI system," and most people have no idea what they're actually holding. He compared it to buying a power saw — a tool that can genuinely hurt you, packaged in a way that implies it cannot.

But here is what neither Gruber nor Willison was focused on, and what Azeem Azhar of Exponential View was: the question of whose interests a personal AI agent actually serves. Azhar's piece this week, titled "Your agent, whose interests?", laid out the problem with unusual precision. When you have a personal AI agent that is persistent, that remembers your preferences, that acts on your behalf — the question of who trained that agent, who controls its objectives, and whose business model it serves becomes not a privacy question but a fiduciary question. Meta's business model is advertising and engagement. An AI agent built on that substrate, one that is making or filtering recommendations on your behalf, carries inside it the priorities of the entity that built it. Not maliciously. Not by design that anyone wrote down in a memo. But structurally, inevitably — the same way a search engine that is paid by advertisers shows you results that serve its advertisers, even when it is genuinely trying to show you the best results for your query.

Now hold that thought. Because what Azhar is describing in the consumer context is about to become an acute professional question for you.

Here is the scenario. It is eighteen months from now. A meaningful number of your clients — let's say the ones between forty and sixty-five, the ones you'd describe as tech-comfortable, the ones who have already adopted voice interfaces and AI tools in their personal lives — those clients have a personal AI agent. Maybe it's Muse. Maybe it's something that came after Muse that is better and more capable. The agent knows their spending patterns, their stated goals, their financial calendar. It is, from their perspective, incredibly helpful. And at some point — because this is what agents do — it starts having opinions about their financial plan. Not loudly. Subtly. It surfaces a question. It prepares them for a meeting with you with talking points drawn from somewhere. It notices a discrepancy between what they told you and what their actual behavior suggests. It recommends they ask you about something it found in a search. Or it quietly pre-processes the recommendation you made last quarter and delivers a verdict on whether it was a good idea, based on data it aggregated from sources you've never seen.

The thing Barron's noted this week, in a piece about how Meta's Muse is triggering concern among brokerage stocks, is that the financial services industry is starting to notice this threat. But the framing in the trade press is about disintermediation — the worry that the agent will replace the adviser. That framing misses the more immediate and more complicated problem. The agent probably will not replace you in eighteen months. What it will do is insert itself between you and your client in a way that changes the conversation before it even starts. Your client will arrive at meetings having already been shaped by a prior AI interaction. They will have questions you didn't prompt. They will have skepticism about things you'd normally explain. Or — and this is the less visible version of the same problem — they will have misplaced confidence about things the agent got wrong.

What this means for trust is hard to overstate. The relationship between an adviser and a client has always rested on a particular kind of information asymmetry: the client trusted that the adviser knew things they didn't, and the adviser trusted that the client would engage honestly. A persistent personal agent disrupts both sides of that equation. The client now has access to a highly confident information source that is available at two in the morning when they're anxious, that never makes them feel embarrassed for asking, and that has a continuous relationship with their data in a way you don't. Whether that source is reliable, whose interests it is actually optimizing for, what it knows about your client that you don't — those are live questions with no clear answers yet.

And here is what this demands from you, if Azhar's framing is right. The fiduciary relationship has to become explicit in a way it has never had to be before. For most of the history of financial planning, the difference between a fiduciary adviser and a non-fiduciary salesperson was a legal distinction that clients understood abstractly, at best. In a world where your client also has an AI agent whose objectives are not fiduciary — whose maker has interests that are not aligned with the client's interests — the fact that your relationship is explicitly and structurally fiduciary becomes one of the most important things you offer. Not as marketing language. As an operational and relational reality that shows up in how you prepare for meetings, how you ask about what else the client has been reading or asking, how you handle the moment when a client walks in with a question that originated in a conversation with their agent and is based on something that is partially right and contextually wrong.

The firms that navigate this well will be the ones that, in twelve to eighteen months, have thought through what it means to have a fiduciary relationship in a world where the client's information environment is no longer controlled by the client or the adviser but by a third party with different interests. That is not a compliance exercise. It is a design challenge. What does your first meeting look like when you assume the client already has an agent? What do your meeting notes capture about what the client has been told by other AI systems? How do your advisers learn to ask the questions that surface the agent's influence — not to undermine it, but to understand what has already shaped the client's thinking? These are not theoretical questions. They are questions that the firms thinking carefully right now will have worked through before the rest of the industry has fully named the problem.

The one-sentence version: when your client's personal AI agent has interests that are not aligned with your client's interests, your explicit fiduciary commitment stops being a legal footnote and starts being your most differentiated product.

Here is the thread that connects both ideas. AI doesn't reduce the value of judgment. It concentrates it. The judgment of the expert who catches the model when it's wrong. The judgment of the firm that knows what it stands for in a world full of agents that stand for something else. In twelve to eighteen months, the firms that thrive will not be the ones that automated the most. They will be the ones that understood what could not be automated, and invested there, deliberately, while everyone else was busy counting efficiency gains.