SearchResearch (10/28/26): How do you know when the AI is wrong? What then?

You can't fact-check everything... 

... there's just too much.  It's the proverbial drinking from a fire hose... even if you're just a dog on the internet.  (Slightly obscure reference.)  

The past few posts have been on a related problem--how to use AI to help find relevant information and check on the accuracy of things.  

But if everything is suspect and you're drowning in content, how do you know when to spend the time and effort on checking?  

You might remember this example from a while ago. Here's the photo I took: 

When I asked Google Lens what this tree was couple of months ago (July, 2026), here's the answer I got... 

I know for certain that this a California Buckeye (Aesculus californica)--you can get all of the details from the authoritative source, CalFlora.  

As I said back then, Google search-by-image (aka Google Lens) gets it wrong. I can't really blame it because it looks a lot like a California Sycamore (Plantanus racemosa) in its fall foliage.  But it's only early July, so that can't be right.  

Even stranger, when I ask Google Lens today (Oct 18, 2026), the answer is different yet again... now it says it's a 

"... a small deciduous tree, which appears to resemble a distressed or stressed Japanese maple (Acer palmatum) or a similar small ornamental tree with thinning, browning foliage."  

That's also very wrong.  

Regular Reader Ramón pointed out that "...you know a lot about trees, so you know that it's wrong... but I don't live in California--how would I know that the answer is wrong?"  

Great question. 

To generalize a bit: When should you trust ANY AI-generated result? 

The teacher in me wants to say "never--just check it all!"  But that's not realistic.  Like the dog in the top image, there's just too much stuff flowing across your screen to check it all.  

Here are four heuristics I've found to help you decide when to turn off the fire hose and pull out the magnifying glass:

1. The "Load-Bearing Fact" Test

Ask yourself: If this fact is wrong, does my entire argument or project collapse? If an AI tells you the capital of North Dakota is Bismarck to settle a trivial debate, a hallucination just costs you pride. But if you are using a specific fact or observation as the foundational premise of a research paper, a syllabus module, or a legal argument, it requires deep verification. Fact-check the structural beams, not every speck on the wallpaper.

2. Missing the Meta-Context 

As demonstrated by the Buckeye masquerading as a fall-foliage of the Sycamore in the middle of summer, AI often ignores the meta-context. 

If an AI generates an answer that conflicts with the physical laws of time, space, or seasonality, dig deeper. Large language models lack common sense about the physical world; when an answer feels disconnected from basic reality—like a pristine winter coat recommendation for a July hike in the Swiss Alps—trust your intuition and verify. (Note that your intuition is based on what YOU bring to the table. The more you know, the better your intuitions will be.)  

3. "Too Neat and Tidy" Warning

Real-world data is messy, contradictory, and full of caveats. AI systems, however, hate that.  They're statistically tuned to be highly agreeable and narrative-driven. If an AI provides an answer that perfectly resolves a notoriously complex debate, or gives a suspiciously clean historical timeline without any friction, it is time to stress-test the source. Reality rarely fits into a perfectly symmetrical box; if the answer reads like a Hollywood script resolving a plot hole, it’s highly suspect. Verify.  

4. The SIFT Trigger: High Emotions Cause Checking

Applying the principles of the SIFT verification method (hat tip to Mike Caulfield), pay attention to your own internal reaction. Does the AI's response immediately validate your deepest held beliefs, or make you unexpectedly outraged? 

Or... When you feel a strong emotional surge of "I knew it!" upon reading a result, that is precisely the moment you need to stop, step back, and triangulate the claim with an authoritative, independent source. When YOU feel completely validated is exactly the moment your guard needs to go up.  Verify that feeling.  

In both cases (very positive or very negative high emotions), you should breathe in and start a bit of checking.  

In short..  

You don't need to fact-check the weather forecast for your afternoon walk. But when the stakes are high, the context feels slightly off, the answer is too perfect, or your emotions spike, that’s your cue to stop reading and start researching.

And, as I've said before--the more you know, the better your intuition will be.  I have pretty good knowledge about plants and trees, so I can smell a suspicious result when it comes over the digital transom.  

Pro tip: To inoculate yourself against AI errors, spend the time to become fairly knowledgeable about the topic.  You need to know enough to know when things seem shady, weird, or just a bit too-perfect.  

Because... the more you know, the more you know.  

Keep searching. 

SearchResearch (10/12/26): How to Verify Assertions in an Age of Infinite Noise

We live in a golden age of information...

Diogènes, the seeker-of-truth. Painting by Jean-Léon Gérôme (1860). P/C Wikimedia.

... and an equally catastrophic era of unverified claims.

Every day, we are bombarded with sleek infographics, AI videos, confident tweets, corporate press releases, and persuasive AI-generated assertions.

They sound authoritative. They look polished. But how often do we stop and ask: Is this actually true, correct, and valid?

In the past couple of weeks, two Regular SearchResearchers (thanks, Ramón and jeffp!) have asked versions of "but how do I know what's true and verifiable?"

It's a great question. So let me try another version of an answer. (I will note that many others have written about this "how to determine.." question as well. See citations below for more perspectives on this.)

In our rush for answers, we often confuse fluency with factuality. A well-written sentence feels true because our brains love cognitive ease.

But the reality is that every day I read assertions that sound authoritative: a statistic in a news story, a quote on social media, a confident paragraph from an AI chatbot. Some are true. Some are almost true. Some are true but don’t support the conclusion they’re attached to. The trouble is that all three look identical on the screen.

So here’s this week’s Challenge:

When someone hands you a claim, how do you decide whether it’s true, correct, and valid?

I’ll give you my answer below, but try think about your own verification routine first. I suspect most people discover they don’t have one. Do you? 

Notice: True, Correct, and Valid are three different questions

We tend to lump them together as “is it right?”, but each one fails in its own way, and each needs a different check.

True, correct, and valid are three different questions

We tend to lump them together as “is it right?”, but each one fails in its own way, and each needs a different check.

	What you’re really asking	How it typically fails
True	Does the claim match the world? Did this thing actually happen, does this person exist, was this said?	Fabricated quotes, misattributed photos, events that never occurred
Correct	Are the details accurate and complete? Right number, right date, right units, right context?	A real statistic from the wrong year; a real quote with the crucial second half cut off; percent vs. percentage points
Valid	Does the conclusion actually follow from the evidence offered?	Correlation sold as causation; one study generalized to everyone; a true fact used to support an unrelated claim

A claim can pass one test and fail the others. “The popularity of the first name Killian is highly correlated with Automotive recalls on air bags” is true and correct, but “kids named Killian causes air bag recalls” is not a valid conclusion. A misremembered or misunderstood statistic might support a perfectly valid argument, but the argument is still built on a wrong number.

A great example of a "spurious correlation."
This website by Tyler Vigen has wonderful (and crazy) examples.

The practical upshot: before you verify anything, decide which of the three you’re checking. Most verification failures I see come from checking the easy one (does a source say this?) and assuming the other two come along for free.

A six-step verification process for claims and assertions

Here’s the routine I actually use. It’s not fast for every claim, but it scales in a reasonable way: a trivial claim takes 30 seconds, a contested one takes an afternoon.  Here’s a good process to follow: 

1. Pin down the claim. Try rewriting it as one precise, checkable sentence. “Coffee is bad for you” isn’t checkable without a major research effort. “Drinking more than 4 cups a day raises heart-attack risk in adults” is. Vague claims can’t be verified, only argued about. Also note who is asserting it and what they want you to do with it. Also notice if the claim is somehow connected to the person making the claim. Do they have an interest in having you believe it?
   
   

2. Stop and notice your reaction. If the claim makes you feel vindicated or outraged, slow down. Strong emotion is the single best signal that you’re about to skip a step or jump to an unwarranted assessment of accuracy.
   
   

3. Trace the evidence upstream. Follow the claim back to its origin: the original study, dataset, transcript, filing, or photo. Every hop between you and the source is a chance for a claim to drift or context to vanish. If you can’t find an origin, that’s a finding in itself.
   
   

4. Read laterally. Most people practice vertical reading—staying on a single page, scrolling up and down, reading an article from top to bottom to judge its credibility. Don’t judge a source just by staring at it in isolation. Open new tabs and see what other people say about the source and the claim. Who runs this site? Who funds this group? What do people with no stake in the outcome say about the claim?
   
   

5. Triangulate. Look for independent confirmation. In particular (as I’ve said many times) find sources that don’t just copy each other. Ten articles citing the same press release are one source, not ten. And look actively for disconfirming evidence.  This is the step everyone skips.
   
   

6. Check the reasoning, then calibrate. Now test validity: does the evidence support this conclusion, or something narrower? Then decide how confident you are, and say so in words: confirmed, likely, unsupported, false. “I couldn’t verify this” is an honest and useful answer.

Steps 1 through 5 mostly test truth and correctness.

Step 6 is where validity lives, and it’s the one that requires thinking rather than searching.

The special case: verifying what an AI tells you

Asking for AI answers break the usual cues. They’re fluent, well-formatted, and uniformly confident, so the usual signals we use to judge a web page tell you nothing.

The routine still works, with a few adjustments:

• Treat every citation as a claim, not as evidence. Open the link. Check that the page exists, says what the AI says it says, and is about the same thing. Mismatched or invented citations are still common.
  
  

• Watch for the plausible detail. AI errors cluster in specifics: dates, names, page numbers, exact figures, quotations. The gist is often right while the details are wrong, which is precisely a correctness failure.
  
  

• Don’t ask the AI to grade itself. “Are you sure?” mostly produces either a cheerful confirmation or a nervous reversal. Neither is verification. Check against an independent source instead.
  
  

• Ask for the upstream source. “Where does this number come from originally?” turns an AI answer into a starting point for step 3 rather than an endpoint.

Used this way, AI is a terrific lead generator and a poor final authority. That’s not a criticism; it’s just recognizing what your tools can do for you. 

 

Teaching this so it sticks

The research here is fairly clear: checklists don’t transfer to students, practiced moves do. Sam Wineburg and Sarah McGrew’s studies at Stanford found that professional fact-checkers outperformed both historians and Stanford undergraduates at judging websites, largely because they left the page quickly and read laterally instead of scrutinizing it. Mike Caulfield’s SIFT method (Stop, Investigate the source, Find better coverage, Trace claims to the original) packages those moves into something teachable.

Here’s what I’ve found works when I teach these skills in a classroom:

• Teach moves, not checklists. Long evaluation checklists (check the URL, the design, the About page...) take ages and reward the wrong things. A handful of quick moves, practiced until they’re habits, beat a 20-item rubric.

• Use real, messy claims. When you teach this skillset, use examples from this week’s feeds, including ones that turn out to be true. If every exercise is a hoax, students learn cynicism, not judgment.

• Separate the three questions explicitly. Give students a claim and have them label each part: what would make this true, correct, valid? Then have them find a claim that passes one test and fails another.

• Think aloud, then hand it over. Model a verification live, narrating your doubts and dead ends. Then let students do one while narrating theirs. The dead ends are the lesson.

• Time-box it. Give a 3-minute limit for a quick check. It forces prioritization and shows that most claims can be triaged fast. It also teaches you that not everything can be done rapidly.

• Grade the reasoning, not the verdict. A student who says “likely, but I couldn’t find the original” with a clear trail has done better work than one who guessed right.

• Close the loop with AI. Have students verify a chatbot’s answer, including its citations. It’s the most memorable lesson in calibrated trust I know. I've used this is my graduate AI classes... and have had students be surprised by the results.

Above all, make verification feel normal and quick, not like a special forensic event. The goal is a reflex: before I share or rely on this, let me take a quick look.

SearchResearch lessons

1. Decide which question you’re answering. True, correct, and valid fail in different ways. Name the one you’re checking.

2. Go upstream. The original source beats any number of summaries of it. If you can’t find the origin, say so.

3. Leave the page. Read laterally to learn what others say about a source before trusting what it says about itself.

4. Count independent sources, not total sources. Ten copies of one press release are one source.

5. Say how sure you are rather than just showing the answer. “Confirmed,” “likely,” and “couldn’t verify” are all legitimate, useful answers, and calibrate the degree of certainty.

Now it’s your turn. What’s your verification routine, and what’s the best lesson you’ve used to teach it? Leave it in the comments. I’ll collect the best ones for a follow-up post.

Keep searching!

--

Citations:

Wineburg, S., & McGrew, S. (2019). Lateral reading and the nature of expertise: Reading less and learning more when evaluating digital information. Teachers College Record, 121(11), 1–40.

Caulfield, M. (2019). Web Literacy for Student Fact-Checkers. Pressbooks.

SearchResearch (9/25, 2026): Decoding the Inscrutable – Using AI to Crack Open Complex Ideas (and Knowing When You've Reached Your Limit)

How does one… 

Gemini's view of how the CRISPR process works to edit DNA in vivo.

… understand the really big, complex, fascinating topics?  

If you’ve been following my explorations here on the blog for a while, you know that my obsession has always been sensemaking—the process by which we gather fragments of information and stitch them together into a coherent mental model. 

And for decades, the dominant paradigm for online sensemaking was keyword search with fancy notetaking: we’d type a few terms, sift through a list of blue links, read the documents, and synthesize the answers in our own heads, maybe using some tools on the side.

But that paradigm is shifting. We are no longer just navigating to information; we are working with and using agentic AI systems and large language models as interactive partners in the synthesis process itself.

Recently, I’ve been thinking a lot about how we use these tools not just to find quick facts, but to learn deeply complex, highly technical concepts that sit far outside our existing domains of expertise. How do you learn something when you don't even know enough to understand the vocabulary?

To explore this, let’s look at one of the most consequential scientific breakthroughs of the 21st century: the CRISPR-Cas9 gene-editing system.  I’ve heard about this for a while and I have a vague understanding of it—but what would be the best way to learn the details?  

The Wall of Jargon: The CRISPR Paper

Suppose you decide you want to truly understand how CRISPR works. You might do the academically rigorous thing and pull up the seminal 2012 Science paper by Jennifer Doudna and Emmanuelle Charpentier: "A Programmable Dual-RNA–Guided DNA Endonuclease in Adaptive Bacterial Immunity."

You read the abstract and hit a brick wall. The language is full terms like crRNA, tracrRNA, endonuclease, protospacer adjacent motif (PAM), and target-site recognition.

If you aren't a molecular biologist, your working memory is instantly overloaded. In the old days, you might have opened twenty browser tabs, individually searching for the definition of each term, trying to hold them all in your head while reading the original sentence. 

It’s a frustrating, high-friction process that often leads to cognitive fatigue and abandonment.  What can we do to make this better?  

The Dialogic Method: Breaking Down Complexity with AI

This is where AI changes the game. Instead of treating an LLM like a glorified encyclopedia, we should treat it like a tireless, infinitely patient tutor. The key is to break the complex idea into interactive, iterative components.

Here are three distinct ways to use AI to penetrate a complex paper like the CRISPR publication:

1. The "Translate and Map" Technique: Instead of asking the AI to "Explain CRISPR," which will just yield a generic Wikipedia-style summary, give the AI the specific, dense text that is blocking you. 

Prompt: [I am reading a biology paper. I have a strong background in computer science but no background in molecular biology. Please translate this specific paragraph about tracrRNA and crRNA into a systems-architecture analogy. What are the 'software' instructions, and what is the 'hardware' execution?" ] 

By anchoring the AI to your existing mental models, it builds a cognitive bridge. Suddenly, the guide RNA isn't abstract biology; it's a search string. The Cas9 protein is a text-editor executing a "find and delete" function.

2. The Socratic Deconstruction When you encounter a complex mechanism, ask the AI to walk you through it step-by-step, but instruct it to pause and wait for your confirmation before moving to the next step. 

Prompt: [ Explain the exact sequence of events when Cas9 binds to DNA. Explain only step one, then wait for me to summarize it back to you before you explain step two. Correct any misunderstandings I have.]

This turns passive reading into active, conversational learning. You are forced to interact with the material, ensuring you actually grasp the base mechanics before layering on more complexity.

3. Parameter Adjusting (The "Dial-a-Level" Approach) If an explanation is still too dense, or perhaps too simple, you can adjust the "resolution" of the information. 

Prompt:  [ You just explained the PAM sequence at a high-school biology level. Now, dial the complexity up to an undergraduate genetics level. Introduce the specific chemical or structural reasons why the Cas9 protein requires this exact sequence to initiate a cut. ]

Epistemic Humility: The Illusion of Knowing

BUT…here’s the most critical part of this new learning method—and the most dangerous trap of the AI era.

When you use an AI to unpack the CRISPR paper using brilliant analogies and perfectly tailored summaries, you will experience a rapid, intoxicating rush of comprehension. Because the AI's prose is so fluent, and because it has successfully connected the concept to your existing knowledge, you will feel a profound sense of mastery.

Do not trust this feeling.

Cognitive psychologists call this the Illusion of Explanatory Depth. 

We frequently overestimate our understanding of complex systems. We think we know how a bicycle or a zipper works until we are handed a piece of paper and asked to draw the mechanism from scratch. Then, suddenly, you don’t know anything.  AI exacerbates this illusion. When the AI does the heavy lifting of synthesis, it feels like your brain did the work.

To be an effective learner, you must practice epistemic humility. 

You must acutely understand the boundaries of your knowledge. Just because you understand the computer-science analogy of CRISPR does not mean you understand the biochemical realities of it. You know the map, but you do not know the territory.

So... How do you establish where your knowledge actually ends? 

One way is to force the AI to test your boundaries with you.  

The Boundary Test Prompt: Here’s a test prompt for you…  

Prompt: [I believe I understand how CRISPR uses guide RNA and the PAM sequence to target and cut specific DNA. I am going to explain it to you in my own words. After I do, I want you to do three things:

1. Tell me where my explanation is factually incorrect.

2. Tell me what critical nuance I am missing.

3. Ask me one challenging question about a scenario where this system might fail or behave unexpectedly, to test if I actually understand the underlying principles.

]

Now... Close those windows you have open with the descriptions of the mechanism. 

Then, when you try to articulate the concept to an empty room, you will quickly discover the gaps in your mental model. The places where you stumble, use vague words like "stuff," or hand-wave a mechanism—those are the boundaries of your knowledge.

Stay on this cycle of question-asking and explaining until you’ve reached the saturation point.  You’ll know it when you get there.  

The Future of Sensemaking

We are moving away from an era of searching for documents and into an era of searching for understanding. The tools at our disposal are miraculous. They allow us to peer into disciplines—like molecular genetics, quantum computing, or macroeconomics—that were previously walled off by decades of specialized jargon.

But using these tools requires a new kind of discipline. We must become expert prompt engineers, not just to get the AI to generate text, but to guide our own cognitive processes. We can use AI to deconstruct the inscrutable, but we must also use it to ruthlessly interrogate our own assumptions. Beware the Illusion of Understanding!  

The goal isn't to know everything. The goal is to deeply explore the complex, expand our mental models, and—perhaps most importantly—retain the wisdom to know exactly what we still don't know.

Keep searching. Keep researching.

SearchResearch (9/11/26): New AI ways of finding hard-to-find things

 I'm sure this has happened to you... 

... you vaguely remember something that you read, possibly from years ago.  You can mostly remember it, but there's no way you'll dredge up the exact reference or the exact details from your memories. You're trying to find exactly the right wire to pull from a tangled mess.  

We all take notes, but if you can't remember enough of the details to search your notes, you might not be able to retrieve that thing that sits there on the edge of memory, a slightly faded dream-vision that's just beyond your grasp.  

Now, of course, we have AI-augmented search. Here's a nice example of one such search I did this week.  

Case in point:  While talking with a colleague at work, I semi-remembered a fascinating paper about how the results of a behavioral study would change significantly if you have an all-female lab assistant crew caring for and tending lab mice versus an all-male crew.  

With Gemini, you could just do a simple description of this half-forgotten result and get the answer back immediately.  Here's what I did, asked a short and sweet question:   

This isn't even an especially coherent statement--it just has all of the parts needed.  ("Testosterone" "male" "lab animal studies" and "profoundly affect the outcome")  

The result was quite good: 

And of course, I clicked on the links (ScienceDaily and Gwern.net) to get to the original source papers. They are: The scent of a man: Gender of experimenter has big impact on rats' stress levels, explains lack of replication of some findings (published in Science Daily, 2014) and  Olfactory exposure to males, including men, causes stress and related analgesia in rodents (published in Nature America, 2014) 

Why is this important? The study sent shockwaves through the clinical research community because it helped explain long-standing issues with the reproducibility of behavioral and pharmacological studies. It demonstrated that something as simple as the gender of the lab assistant could profoundly alter experimental outcomes, leading many scientific journals and labs to subsequently mandate the reporting of experimenter sex in behavioral studies.

More generally.. .One of the consequences of fast/cheap/easy search tools is that you can dig more deeply into the material than was plausible before.  Do so!  

For instance, I'm now re-reading East of Eden, by John Steinbeck (first time since high school). I found that I've forgotten most of the story, and I find myself constantly stumbling over terms and concepts that I'm sure I didn't know in high school, and almost certainly never looked up at the time.  Words like timshel or paregoric just slipped beneath my radar.  But being able to quickly dive more deeply into these concepts certainly makes the rest of the story more meaningful.  

Search is awfully handy, and AI-based search lets you pull out exactly the right wire you've been looking for in the rat's nest of mental cabling.  

SearchResearch Lessons 

1. You can often retrieve even vaguely remembered articles with a straightforward AI search.  The cost and time of tracking down important ideas is very low--do it, rather than relying solely on fallible memory.  

2. IMPORTANT: Sometimes the AI can get it wrong--you still need to verify the finding.  

3. Don't over-specify the details of the thing you seek.  Note that I did not say "in mice" in my query, but just left it as the more generic "lab animals."  

Keep searching.  

SearchResearch (8/26/26): What ELSE can your AI do? (Knowing what's possible--the value of a mental model)

The last couple of posts... 

... have been about learning what ELSE your AI system can do.  We talked about file conversions you didn't know it could do, remarkable summaries of email threads, converting YouTube videos into blog posts, doing tree ring analysis from a photo, generating diagrams, etc etc.  

As you probably noticed, there's a lot of things that you didn't know your AI could do. 

Let me show you one more that I used this week, then talk about the general problem--how can you know what else your AI can do.  

Story:  I was setting up a meeting on my Google Calendar when I noticed that the information on my popup was seriously out of date.  

My Google Calendar popup looked like this:  

Thing is, I don't use Picasa any more, I don't have a Twitter profile, etc.  

I want to change that information.  There's just one problem: WHAT is this thing called?  How DO I change that information?  

Searching for [Google Calendar popup] isn't a great search.  What to do?  

Easy.  I just did a Gemini by uploading the image and asking: 

      [how do I change the information in this red box?]  

Sure enough.. it told me--easy, simple, direct.  

Basically it said: 

"Go directly to myaccount.google.com/profile (or navigate to Google Account > Personal info > scroll down to "Choose what others see" / "About me")." Edit.

The huge advantage of this show-it-the-picture method is that I didn't need to know any specialized terminology.  It was able to figure out the context from the image itself. 

Remember this trick next time you have to fix something for which words fail you. 

(I've used this same trick to fix refrigerators and dishwashers, as well as software!)  

Here's the point: Why you need to know what your AI can / cannot do

Every tool (and every AI) carries an implicit contract about what it will do when you use it. This is what we've been calling a mental model: that is, the beliefs a person holds about a system's structure and behavior, which they use to predict what will happen next. [Norman 1983] 

Note that mental models do not have to be accurate to be useful--but they must be accurate about the boundaries, because that is where prediction fails and errors escape review.

The boundaries define what you know about an AI.  

For AI systems, those boundaries are unusually hard to see and understand because they're complicated.  (See the diagram at the top of this post.)  

As we've discussed, the term "jagged technological frontier" is used to describe AI capabilities that are unevenly scattered across tasks of seemingly similar difficulty. That phrase has been popularized (often shortened to the "jagged edge" regarding AI) by Ethan Mollick in a number of articles.  AI tasks that seem like they might be equally difficult, often are NOT.  It's difficult to predict what an AI can do vs. what it can't do.  This is one kind of jaggedness.   

Here's some data for you: In a preregistered field experiment with 758 Boston Consulting Group consultants, subjects using GPT-4 on eighteen tasks inside the frontier completed 12.2% more tasks, 25.1% faster, at measurably higher quality. [Dell'Acqua, 2026]

On a single managerial task deliberately placed outside it, the same tool made consultants 19% less likely to reach the correct answer. Nothing in the task's description told you which side of the line it fell on. Easy?  Or impossible?  

Notice the asymmetry: inside the frontier, AI acts as an accelerator; outside, it acts as a blind spot that actively degrades performance.

Case one: when knowing an AI capability pays. Consider a developer who has correctly modeled LLM code assistants as strong on well-specified, convention-heavy, immediately testable code. They really, truly understand what it can do. 

In a controlled trial, developers asked to implement an HTTP server in JavaScript finished 55.8% faster with GitHub Copilot than without. [Peng et al. 2023] 

That is, the task fit the tool's shape: bounded scope, dense boilerplate, and — critically — an oracle. The output could be run. A practitioner who knows this reaches for the tool on scaffolding and refactors, and reclaims hours that would otherwise go to typing what the model already knows.

Case two: not knowing an AI limitation costs real money. In Mata v. Avianca, counsel submitted a brief citing six judicial opinions that ChatGPT had invented, then produced fabricated excerpts when challenged; Judge Castel imposed a $5,000 Rule 11 sanction. [S.D.N.Y. 2023] The failure was not the hallucination — it was the missing model. Dahl et al. tested public LLMs on verifiable questions about federal cases and found hallucination rates between 58% and 88%, with models frequently unable to flag their own fabrications and prone to accepting a user's false legal premises [Dahl et al. 2024] The thing is,  case retrieval seems like the sort of thing a language model should do well. It is not, and the user had no way to know that from the interface.

The perception gap compounds the problem. In a randomized trial of experienced open-source developers on their own repositories, AI assistance increased task completion time by 19% — while participants, after the fact, estimated it had made them 20% faster. [Becker et al. 2025] That is, even skilled practitioners cannot reliably feel where the frontier lies. 

This is the misuse–disuse dynamic described for automation generally: performance degrades both when operators over-trust a system beyond its competence and when they abandon a competent one. [Parasuraman and Riley 1997]

The professional implication is procedural, not attitudinal. 

Your understanding of the AI's capability must be established empirically, per task and per model version, against ground truth you hold independently — and re-established when the version changes. Knowing what your AI cannot do is not a caveat on its value. It is the precondition for it.

And, as we've seen, just because one AI can do something tells you nothing about whether or not another AI can do the same thing.  The frontier between AI systems is as complicated and unpredictable as it is within an AI.  

Keep searching! 

---

References

Becker, J., N. Rush, E. Barnes, and D. Rein. 2025. "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." arXiv:2507.09089. https://arxiv.org/abs/2507.09089

Dahl, M., V. Magesh, M. Suzgun, and D. E. Ho. 2024. "Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models." Journal of Legal Analysis 16(1): 64–93. https://doi.org/10.1093/jla/laae003

Dell'Acqua, F., E. McFowland III, E. Mollick, H. Lifshitz, K. C. Kellogg, S. Rajendran, L. Krayer, F. Candelon, and K. R. Lakhani. 2026. "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality." Organization Science 37(2): 403–423. https://doi.org/10.1287/orsc.2025.21838

Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. June 22, 2023) (No. 22-cv-1461, Castel, J.).

Norman, D. A. 1983. "Some Observations on Mental Models." In Mental Models, ed. D. Gentner and A. L. Stevens, 7–14. Hillsdale, NJ: Lawrence Erlbaum.

Parasuraman, R., and V. Riley. 1997. "Humans and Automation: Use, Misuse, Disuse, Abuse." Human Factors 39(2): 230–253. https://doi.org/10.1518/001872097778543886

Peng, S., E. Kalliamvakou, P. Cihon, and M. Demirer. 2023. "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot." arXiv:2302.06590. https://arxiv.org/abs/2302.06590