The last couple of posts...
... have been about learning what ELSE your AI system can do. We talked about file conversions you didn't know it could do, remarkable summaries of email threads, converting YouTube videos into blog posts, doing tree ring analysis from a photo, generating diagrams, etc etc.
As you probably noticed, there's a lot of things that you didn't know your AI could do.
Let me show you one more that I used this week, then talk about the general problem--how can you know what else your AI can do.
Story: I was setting up a meeting on my Google Calendar when I noticed that the information on my popup was seriously out of date.
My Google Calendar popup looked like this:
Thing is, I don't use Picasa any more, I don't have a Twitter profile, etc.
I want to change that information. There's just one problem: WHAT is this thing called? How DO I change that information?
Searching for [Google Calendar popup] isn't a great search. What to do?
Easy. I just did a Gemini by uploading the image and asking:
[how do I change the information in this red box?]
Sure enough.. it told me--easy, simple, direct.
Basically it said:"Go directly to myaccount.google.com/profile (or navigate to Google Account > Personal info > scroll down to "Choose what others see" / "About me")." Edit.
Here's the point: Why you need to know what your AI can / cannot do
Every tool (and every AI) carries an implicit contract about what it will do when you use it. This is what we've been calling a mental model: that is, the beliefs a person holds about a system's structure and behavior, which they use to predict what will happen next. [Norman 1983]
Note that mental models do not have to be accurate to be useful--but they must be accurate about the boundaries, because that is where prediction fails and errors escape review.
The boundaries define what you know about an AI.
For AI systems, those boundaries are unusually hard to see and understand because they're complicated. (See the diagram at the top of this post.)
As we've discussed, the term "jagged technological frontier" is used to describe AI capabilities that are unevenly scattered across tasks of seemingly similar difficulty. That phrase has been popularized (often shortened to the "jagged edge" regarding AI) by Ethan Mollick in a number of articles. AI tasks that seem like they might be equally difficult, often are NOT. It's difficult to predict what an AI can do vs. what it can't do. This is one kind of jaggedness.
Here's some data for you: In a preregistered field experiment with 758 Boston Consulting Group consultants, subjects using GPT-4 on eighteen tasks inside the frontier completed 12.2% more tasks, 25.1% faster, at measurably higher quality. [Dell'Acqua, 2026]
On a single managerial task deliberately placed outside it, the same tool made consultants 19% less likely to reach the correct answer. Nothing in the task's description told you which side of the line it fell on. Easy? Or impossible?
Notice the asymmetry: inside the frontier, AI acts as an accelerator; outside, it acts as a blind spot that actively degrades performance.
Case one: when knowing an AI capability pays. Consider a developer who has correctly modeled LLM code assistants as strong on well-specified, convention-heavy, immediately testable code. They really, truly understand what it can do.
In a controlled trial, developers asked to implement an HTTP server in JavaScript finished 55.8% faster with GitHub Copilot than without. [Peng et al. 2023]
That is, the task fit the tool's shape: bounded scope, dense boilerplate, and — critically — an oracle. The output could be run. A practitioner who knows this reaches for the tool on scaffolding and refactors, and reclaims hours that would otherwise go to typing what the model already knows.
Case two: not knowing an AI limitation costs real money. In Mata v. Avianca, counsel submitted a brief citing six judicial opinions that ChatGPT had invented, then produced fabricated excerpts when challenged; Judge Castel imposed a $5,000 Rule 11 sanction. [S.D.N.Y. 2023] The failure was not the hallucination — it was the missing model. Dahl et al. tested public LLMs on verifiable questions about federal cases and found hallucination rates between 58% and 88%, with models frequently unable to flag their own fabrications and prone to accepting a user's false legal premises [Dahl et al. 2024] The thing is, case retrieval seems like the sort of thing a language model should do well. It is not, and the user had no way to know that from the interface.
The perception gap compounds the problem. In a randomized trial of experienced open-source developers on their own repositories, AI assistance increased task completion time by 19% — while participants, after the fact, estimated it had made them 20% faster. [Becker et al. 2025] That is, even skilled practitioners cannot reliably feel where the frontier lies.
This is the misuse–disuse dynamic described for automation generally: performance degrades both when operators over-trust a system beyond its competence and when they abandon a competent one. [Parasuraman and Riley 1997]
The professional implication is procedural, not attitudinal.
Your understanding of the AI's capability must be established empirically, per task and per model version, against ground truth you hold independently — and re-established when the version changes. Knowing what your AI cannot do is not a caveat on its value. It is the precondition for it.
And, as we've seen, just because one AI can do something tells you nothing about whether or not another AI can do the same thing. The frontier between AI systems is as complicated and unpredictable as it is within an AI.
Keep searching!
References
Becker, J., N. Rush, E. Barnes, and D. Rein. 2025. "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." arXiv:2507.09089. https://arxiv.org/abs/2507.09089
Dahl, M., V. Magesh, M. Suzgun, and D. E. Ho. 2024. "Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models." Journal of Legal Analysis 16(1): 64–93. https://doi.org/10.1093/jla/laae003
Dell'Acqua, F., E. McFowland III, E. Mollick, H. Lifshitz, K. C. Kellogg, S. Rajendran, L. Krayer, F. Candelon, and K. R. Lakhani. 2026. "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality." Organization Science 37(2): 403–423. https://doi.org/10.1287/orsc.2025.21838
Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. June 22, 2023) (No. 22-cv-1461, Castel, J.).
Norman, D. A. 1983. "Some Observations on Mental Models." In Mental Models, ed. D. Gentner and A. L. Stevens, 7–14. Hillsdale, NJ: Lawrence Erlbaum.
Parasuraman, R., and V. Riley. 1997. "Humans and Automation: Use, Misuse, Disuse, Abuse." Human Factors 39(2): 230–253. https://doi.org/10.1518/001872097778543886
Peng, S., E. Kalliamvakou, P. Cihon, and M. Demirer. 2023. "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot." arXiv:2302.06590. https://arxiv.org/abs/2302.06590
No comments:
Post a Comment