SearchResearch (8/26/26): What ELSE can your AI do? (Knowing what's possible--the value of a mental model)

The last couple of posts... 

... have been about learning what ELSE your AI system can do.  We talked about file conversions you didn't know it could do, remarkable summaries of email threads, converting YouTube videos into blog posts, doing tree ring analysis from a photo, generating diagrams, etc etc.  

As you probably noticed, there's a lot of things that you didn't know your AI could do. 

Let me show you one more that I used this week, then talk about the general problem--how can you know what else your AI can do.  

Story:  I was setting up a meeting on my Google Calendar when I noticed that the information on my popup was seriously out of date.  

My Google Calendar popup looked like this:  

Thing is, I don't use Picasa any more, I don't have a Twitter profile, etc.  

I want to change that information.  There's just one problem: WHAT is this thing called?  How DO I change that information?  

Searching for [Google Calendar popup] isn't a great search.  What to do?  

Easy.  I just did a Gemini by uploading the image and asking: 

      [how do I change the information in this red box?]  

Sure enough.. it told me--easy, simple, direct.  

Basically it said: 

"Go directly to myaccount.google.com/profile (or navigate to Google Account > Personal info > scroll down to "Choose what others see" / "About me")." Edit.

The huge advantage of this show-it-the-picture method is that I didn't need to know any specialized terminology.  It was able to figure out the context from the image itself. 

Remember this trick next time you have to fix something for which words fail you. 

(I've used this same trick to fix refrigerators and dishwashers, as well as software!)  

Here's the point: Why you need to know what your AI can / cannot do

Every tool (and every AI) carries an implicit contract about what it will do when you use it. This is what we've been calling a mental model: that is, the beliefs a person holds about a system's structure and behavior, which they use to predict what will happen next. [Norman 1983] 

Note that mental models do not have to be accurate to be useful--but they must be accurate about the boundaries, because that is where prediction fails and errors escape review.

The boundaries define what you know about an AI.  

For AI systems, those boundaries are unusually hard to see and understand because they're complicated.  (See the diagram at the top of this post.)  

As we've discussed, the term "jagged technological frontier" is used to describe AI capabilities that are unevenly scattered across tasks of seemingly similar difficulty. That phrase has been popularized (often shortened to the "jagged edge" regarding AI) by Ethan Mollick in a number of articles.  AI tasks that seem like they might be equally difficult, often are NOT.  It's difficult to predict what an AI can do vs. what it can't do.  This is one kind of jaggedness.   

Here's some data for you: In a preregistered field experiment with 758 Boston Consulting Group consultants, subjects using GPT-4 on eighteen tasks inside the frontier completed 12.2% more tasks, 25.1% faster, at measurably higher quality. [Dell'Acqua, 2026]

On a single managerial task deliberately placed outside it, the same tool made consultants 19% less likely to reach the correct answer. Nothing in the task's description told you which side of the line it fell on. Easy?  Or impossible?  

Notice the asymmetry: inside the frontier, AI acts as an accelerator; outside, it acts as a blind spot that actively degrades performance.

Case one: when knowing an AI capability pays. Consider a developer who has correctly modeled LLM code assistants as strong on well-specified, convention-heavy, immediately testable code. They really, truly understand what it can do. 

In a controlled trial, developers asked to implement an HTTP server in JavaScript finished 55.8% faster with GitHub Copilot than without. [Peng et al. 2023] 

That is, the task fit the tool's shape: bounded scope, dense boilerplate, and — critically — an oracle. The output could be run. A practitioner who knows this reaches for the tool on scaffolding and refactors, and reclaims hours that would otherwise go to typing what the model already knows.

Case two: not knowing an AI limitation costs real money. In Mata v. Avianca, counsel submitted a brief citing six judicial opinions that ChatGPT had invented, then produced fabricated excerpts when challenged; Judge Castel imposed a $5,000 Rule 11 sanction. [S.D.N.Y. 2023] The failure was not the hallucination — it was the missing model. Dahl et al. tested public LLMs on verifiable questions about federal cases and found hallucination rates between 58% and 88%, with models frequently unable to flag their own fabrications and prone to accepting a user's false legal premises [Dahl et al. 2024] The thing is,  case retrieval seems like the sort of thing a language model should do well. It is not, and the user had no way to know that from the interface.

The perception gap compounds the problem. In a randomized trial of experienced open-source developers on their own repositories, AI assistance increased task completion time by 19% — while participants, after the fact, estimated it had made them 20% faster. [Becker et al. 2025] That is, even skilled practitioners cannot reliably feel where the frontier lies. 

This is the misuse–disuse dynamic described for automation generally: performance degrades both when operators over-trust a system beyond its competence and when they abandon a competent one. [Parasuraman and Riley 1997]

The professional implication is procedural, not attitudinal. 

Your understanding of the AI's capability must be established empirically, per task and per model version, against ground truth you hold independently — and re-established when the version changes. Knowing what your AI cannot do is not a caveat on its value. It is the precondition for it.

And, as we've seen, just because one AI can do something tells you nothing about whether or not another AI can do the same thing.  The frontier between AI systems is as complicated and unpredictable as it is within an AI.  

Keep searching! 

---

References

Becker, J., N. Rush, E. Barnes, and D. Rein. 2025. "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." arXiv:2507.09089. https://arxiv.org/abs/2507.09089

Dahl, M., V. Magesh, M. Suzgun, and D. E. Ho. 2024. "Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models." Journal of Legal Analysis 16(1): 64–93. https://doi.org/10.1093/jla/laae003

Dell'Acqua, F., E. McFowland III, E. Mollick, H. Lifshitz, K. C. Kellogg, S. Rajendran, L. Krayer, F. Candelon, and K. R. Lakhani. 2026. "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality." Organization Science 37(2): 403–423. https://doi.org/10.1287/orsc.2025.21838

Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. June 22, 2023) (No. 22-cv-1461, Castel, J.).

Norman, D. A. 1983. "Some Observations on Mental Models." In Mental Models, ed. D. Gentner and A. L. Stevens, 7–14. Hillsdale, NJ: Lawrence Erlbaum.

Parasuraman, R., and V. Riley. 1997. "Humans and Automation: Use, Misuse, Disuse, Abuse." Human Factors 39(2): 230–253. https://doi.org/10.1518/001872097778543886

Peng, S., E. Kalliamvakou, P. Cihon, and M. Demirer. 2023. "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot." arXiv:2302.06590. https://arxiv.org/abs/2302.06590

SearchResearch (8/11/26): What can your AI system do? (2/2)

Knowing what your tools can do... 

... is an important bit of knowledge if you're going to be using sophisticated tools. Taking a cue from yesterday's post, your mental model of the tools you have needs to be fairly accurate, predictive, and in these days of constant and continual change, fairly up-to-date.  

Here are a few things that you MIGHT not know your LLM can do.  

1. Filetype conversions:  

Did you know that some LLMs can convert some files?  

ChatGPT and Claude - can convert: m4a files to mp3; JPG to PNG; MP4 to animated GIF; WAV to MP3... etc etc.  (FWIW, Gemini currently cannot do any of these conversions.)  Just upload a file to Claude or ChatGPT and then say some version of: 

  [convert this file to PNG] 

Here's an example using Claude to convert a JPG to a GIF.  

Note that you can also use this trick to convert video files into animated GIFs.  (This is my most common use case!)  

2. Summaries of current topics:  

ChatGPT - create and email a periodic summaries on topics of interest. (See their documentation here: Scheduled tasks in ChatGPT)  Here's how I set up one for me on a few topics of interest.   

And the result that gets sent to your email looks like this with each section supplied with the latest info scraped from the web and summarized for you:  

To re-find all of your emails from ChatGPT, search for [from:chatgpt] in your Gmail search bar.  

3. Transform YouTube videos into blog posts: 

Gemini and ChatGPT will let you a YouTube URL to extract transcripts, build chapter breakdowns, or completely rewrite the video content into a formatted blog post.  Here's an example:  

[please convert this youtube video into a blog post for me  https://www.youtube.com/watch?v=emc742B-Llc ]

A nice summary of the existing video content with links to particular points in the video. This is useful when you find a video that you need to search purely for information purposes and you don’t feel like sitting through 32 minutes to get one bit of information. (You could look at the transcript and do a search, but this actually gets you to that place AND shows you the visual context. Incredibly handy.)  You could also say something like “give me a short bulleted summary of the high points.”  OR.. If you’ve got something super-particular you’re seeking, you can also just ask Gemini about it: 

“What does this person say about HCI?”

“Does the interview reveal anything about possible failures of UX design in AI systems?”

You get the idea. 

4. Email summary analysis:   

Gemini - Try this query in regular Gemini:

 [Based on my emails and calendar, what are the main things
   competing for my attention this week?]  

Of course, you'll have to be logged into your Google account for this to work.  

Here's what the output looks like (with names blurred out). You can see it picked up on a few things that I still need to do and noticed a conflict on Friday between 2 meetings.  

Notice the helpful suggested action item buttons at the bottom.  Clicking on them creates draft responses that were correct and helpful.  

(FWIW, I tried this in Claude, but got stuck in an endless cycle of trying to give permission to Claude for my calendar access.  I gave up after a while.)  

5. Photo summary analysis:  

Again with Gemini... 

 [ What patterns do you notice in my photos from the last month? ] 

And below this was a selection of 4 images that were fairly representative of what I did this month and the people I was hanging around with during the first two weeks of August.  

SearchResearch Lessons

The deep point of this post is that an important part of your mental model of AIs is knowing what it can do.  

That is, what is your AI capable of doing for you? 

A couple of things worth noticing: 

1. What the AIs can do keeps changing with each version update.  It's true that Gemini can't (or won't) do filetype conversions... but maybe the next version will.  You have to keep track of the capabilities not just by company, but also by version. 

2. The AIs will also LOSE capabilities.  This happens with search engines, photo editing apps... and it will happen with AIs as well. Don't be surprised when this happens.  (Just remember that I told you so...)  

3. Capabilities also sometimes drop out for a while.  AI as a service lives on the bleeding and jagged edge of capability. Behaviors that you count on might just disappear for a while and then come back.  It's not your fault, but usually the side effect of some deep technical issue way inside the guts of your AI.  As Douglas Adams would say, Don't Panic, but search for another way to do the same thing.  

So.. you need to stay up on what works and what doesn't work.  

If you've found a surprising capability in one of your favorite AIs, let us know by leaving a comment in the thread. Knowledge is best when shared! 

Stay tuned. 

Keep searching.  

SearchResearch 8/10/6: What can your AI system do? (1/2)

 When you use a tool, 

... you usually know what it can do. A knife cuts, a hammer drives in nails, a pen draws a line.  

Of course, tools can be specialized for particular tasks (e.g., a fish knife or a ball peen hammer), or can be floridly multi-purpose (like your favorite computer or a Swiss Army Knife like that above).  

This raises a great question about what's your mental model of an AI system?  

Classically, AI systems were fairly specific in what they could do.  In the great AI expert systems of the past, they could do exactly one thing and do it very, very well. But with the rise of modern AI, especially LLMs, things have changed, and now AI systems can do a wide variety of tasks. 

For us, the question is: what are those tasks? 

This is the problem of knowing what's possible with an AI. 

It's pretty clear that our AI systems can do a number of things: translate languages; solve logic problems; answer deep questions; make interesting scientific discoveries; create wonderful visualizations of information.  

But maybe that fancy new AI system has a chat feature that lets me chat with the ghost of my aged Grandmother, gone for twenty years.  How would you know if it could or not?  

People have written about the "jagged edge of AI" as being a difference in the capability of a single AI system.  (For details, see "Navigating the jagged technological frontier.") Your AI can do some complicated things really well but then it will fail on similar tasks that are fairly straightforward, giving rise to the sense that the edge of competence is very jagged. 

That's unfortunate because it's tough to tell what your AI will be able to do. 

But there's also a jagged edge between different AI systems. That is, just because one AI system can do a particular thing, it's unclear that another, different AI can do the same thing.  

Here's an example... figuring out how old a tree is.  

While out on a hike this weekend, I came across an exposed tree stump that had been cut down. It had shown a bunch of tree rings, and I was wondering how old the tree was. This, I thought, was the perfect application for an AI.

So I gave three popular AI systems the task of counting the number of tree rings on a stump. As you can see the results between the three AI systems are very, very different.  I prompted each by uploading the image above and then giving it the task of [how old is this tree] letting it figure out the method by which it should count tree rings. 

Claude (Opus 5, medium) did a reasonably good job that looks very impressive but it's not actually very accurate even though it gives the impression of high accuracy. The response has lots of technical language and botanical terms. The image analysis looks very precise, grainy, and scientific. The whole this is impressive... but wrong.  

ChatGPT 5.6 did a pretty good job, although if you look carefully you'll see that the numbers of the tree rings are not quite right.  Things don't line up the way they should.

Gemini did a pretty terrible job. If you look at the numbers on the tree rings, you can see that the zero is actually out about five years after the tree started to grow. If you look carefully there are rings that are labeled both 5 and 10 or if you look a little bit further out, there are rings labeled both 30 and 25. 

What's up with that?  

(Interestingly, the first time I asked Gemini for the rings on the image, it said it created the image, but didn't REALLY.  But by asking for it to re-do the image, it coughed up this image, but it also changed its initial estimate of 52 years old to 68. I don't think it's being sycophantic, just nondeterministic!)  

However... just because I'm that kind of guy, I stood there and counted the tree rings manually.  Counting carefully I found 85, but I agree with Claude that outer rings are difficult to determine.

In other words, they're all wrong by different amounts.

I know this is a tough task, but a good AI should recognize the boundary of what it can do... and sussing out the details of tree rings from a medium-resolution image should trigger caution. 

This is huge variation across AIs on the same task.  We know this, and it's not a huge surprise... but it is a kind of jagged edge. 

Let's look at a different kind of problem--
     [generate an exploded parts diagram of a modern racing bicycle] 

Here are the results: 

Gemini: 

Gemini's version of a bicycle, not bad, but also NOT exploded.

ChatGPT: 

ChatGPT's bike is more exploded that Gemini's. 

Claude: 

Claude's diagram is fairly... basic... exploded, but wrong in so many ways.  

The short story is that AI capabilities vary a LOT between systems as well.  The jagged edge is everywhere.

And the key question for us is this: How can we have useful mental models of AI systems when capabilities vary SO much?? 

In the next post, we'll talk about understanding what your AI system is good at doing... and why knowing what it's BAD at doing is just as important a part of your mental model. 

Keep searching!