SearchResearch 8/10/6: What can your AI system do? (1/2)

 When you use a tool, 

... you usually know what it can do. A knife cuts, a hammer drives in nails, a pen draws a line.  

Of course, tools can be specialized for particular tasks (e.g., a fish knife or a ball peen hammer), or can be floridly multi-purpose (like your favorite computer or a Swiss Army Knife like that above).  

This raises a great question about what's your mental model of an AI system?  

Classically, AI systems were fairly specific in what they could do.  In the great AI expert systems of the past, they could do exactly one thing and do it very, very well. But with the rise of modern AI, especially LLMs, things have changed, and now AI systems can do a wide variety of tasks. 

For us, the question is: what are those tasks? 

This is the problem of knowing what's possible with an AI. 

It's pretty clear that our AI systems can do a number of things: translate languages; solve logic problems; answer deep questions; make interesting scientific discoveries; create wonderful visualizations of information.  

But maybe that fancy new AI system has a chat feature that lets me chat with the ghost of my aged Grandmother, gone for twenty years.  How would you know if it could or not?  

People have written about the "jagged edge of AI" as being a difference in the capability of a single AI system.  (For details, see "Navigating the jagged technological frontier.") Your AI can do some complicated things really well but then it will fail on similar tasks that are fairly straightforward, giving rise to the sense that the edge of competence is very jagged. 

That's unfortunate because it's tough to tell what your AI will be able to do. 

But there's also a jagged edge between different AI systems. That is, just because one AI system can do a particular thing, it's unclear that another, different AI can do the same thing.  

Here's an example... figuring out how old a tree is.  

While out on a hike this weekend, I came across an exposed tree stump that had been cut down. It had shown a bunch of tree rings, and I was wondering how old the tree was. This, I thought, was the perfect application for an AI.

So I gave three popular AI systems the task of counting the number of tree rings on a stump. As you can see the results between the three AI systems are very, very different.  I prompted each by uploading the image above and then giving it the task of [how old is this tree] letting it figure out the method by which it should count tree rings. 

Claude (Opus 5, medium) did a reasonably good job that looks very impressive but it's not actually very accurate even though it gives the impression of high accuracy. The response has lots of technical language and botanical terms. The image analysis looks very precise, grainy, and scientific. The whole this is impressive... but wrong.  

ChatGPT 5.6 did a pretty good job, although if you look carefully you'll see that the numbers of the tree rings are not quite right.  Things don't line up the way they should.

Gemini did a pretty terrible job. If you look at the numbers on the tree rings, you can see that the zero is actually out about five years after the tree started to grow. If you look carefully there are rings that are labeled both 5 and 10 or if you look a little bit further out, there are rings labeled both 30 and 25. 

What's up with that?  

(Interestingly, the first time I asked Gemini for the rings on the image, it said it created the image, but didn't REALLY.  But by asking for it to re-do the image, it coughed up this image, but it also changed its initial estimate of 52 years old to 68. I don't think it's being sycophantic, just nondeterministic!)  

However... just because I'm that kind of guy, I stood there and counted the tree rings manually.  Counting carefully I found 85, but I agree with Claude that outer rings are difficult to determine.

In other words, they're all wrong by different amounts.

I know this is a tough task, but a good AI should recognize the boundary of what it can do... and sussing out the details of tree rings from a medium-resolution image should trigger caution. 

This is huge variation across AIs on the same task.  We know this, and it's not a huge surprise... but it is a kind of jagged edge. 

Let's look at a different kind of problem--
     [generate an exploded parts diagram of a modern racing bicycle] 

Here are the results: 

Gemini: 

Gemini's version of a bicycle, not bad, but also NOT exploded.

ChatGPT: 

ChatGPT's bike is more exploded that Gemini's. 

Claude: 

Claude's diagram is fairly... basic... exploded, but wrong in so many ways.  

The short story is that AI capabilities vary a LOT between systems as well.  The jagged edge is everywhere.

And the key question for us is this: How can we have useful mental models of AI systems when capabilities vary SO much?? 

In the next post, we'll talk about understanding what your AI system is good at doing... and why knowing what it's BAD at doing is just as important a part of your mental model. 

Keep searching!

SearchResearch (07/31/26): Mental models of data needed as well

What do you believe about the data?

In the previous post I wrote about mental models of systems—that is, how you believe an information retrieval tool or web search system operates. Is it okay that you think it's all hamsters all the way down?  

But the other side of the coin is your mental model of what data the system is operating with.   Tool operation is important, but tools operate on something, and understanding the properties of something is just as important.  

Let me ask you, “what is your mental model of the information that search tools operate on?”  

Think of it along 3 dimensions:  

1. Completeness: How much of the total information space out in the world is available to you by search?  Is it complete?

2. Kinds: What different KINDS of information can be found? 

3. Origin: Where does the information come from?  

In my research I’ve found that searchers usually have a relatively limited mental model of the information space.  As you can extrapolate from the previous post, people think of Google as working on either a GIANT library of content, or a GIANT database with structured content in it.  

The truth is, neither is true.  Google indexes all of the web pages it can reach by crawling (following links from one page to another) OR data that it gets from specialized feeds (like news from publishers).  Google Scholar is a great example of both—they crawl the web looking for texts that look like scientific papers, and they get data dumps from specific publishers. There are also content sites that Google maintains like YouTube for videos or Google Play for books, apps, and games. 

The entire corpus of human knowledge is NOT what Google is about. 
It's a sprawling, complicated mess.  
P/C by Gemini.

As a consequence, people believe that everything—all of the knowledge of humanity—is indexed and available with a simple search.  During my studies I’ve had multiple people tell me that “everything is in Google.”  (You can read more about this in the paper In Google We Trust: User’s decisions on rank, position, and relevance.)  

This research result also tells us that users exhibit a strong "trust bias" toward search engine rankings. Participants consistently assumed that if a page was at the top of Google, it was the definitive answer. Users frequently assume that if something cannot be found on Google, it does not exist. 

For more insight on this, see: Transparent queries: investigate users' mental models of search engines

We know that searchers operate with simplified mental models. They tend to view search engines not as specific, curated index databases with technical limits, but as direct windows onto the entire universe of human knowledge. 

Unfortunately, that’s not even close to being true.  People assume Google crawls 100% of the live web in real time. In reality, Google’s index represents a tiny fraction of the total web content (often estimated at less than 5%–10% when accounting for the Deep Web).

What’s more, in actual use, searchers tend to use the tools they have in fairly shallow ways.  We’ve known for a while that the vast majority of users never look beyond the first page of results (often called the "principle of least effort"). Users unconsciously equate unindexed or low-ranked with non-existent. (See: "Search engine use behavior of students and faculty”)

As a consequence, searchers often believe that If a query yields no results, they tend to infer that the object, fact, or paper does not exist anywhere, rather than recognizing that Google's crawlers simply haven't discovered, indexed, or rendered it.

People also frequently expect Google to retain historical versions of every page forever, confusing web indexing with digital archiving (like the Internet Archive’s Wayback Machine).

What Google Can Find

1. Crawlable Surface Web Content: HTML pages, publicly accessible blogs, news, public forums, and open-access documents.

2. Specific Indexed Document Types: PDF, DOCX, PPT, XLSX, TXT, EPUB, and specialized formats.  (But also note that the site: operator will find a file with that extension no matter what the extension is.  Example: [filetype:randomness] – this isn’t a real file type extension, but Google will find documents with this extension!)  

3. Structured Data & Knowledge Graph Entities: Information formatted via Schema.org (recipes, events, products, flight statuses, explicit facts about known entities). Often imported directly rather than crawled. 

4. Media Metadata: Images (via ALT text, surroundings, and computer vision), videos (via transcription and timestamps), and audio metadata.

5. Renderable Dynamic Content: such as JavaScript-rendered pages (Client-side rendering, React/Vue applications) that the Googlebot can execute.

What Google Cannot (or Does Not) Find

A. The Deep Web (walled-garden Content): Information behind paywalls, subscription databases (e.g., academic databases like JSTOR, LexisNexis), private intranets, or user login screens.

B. Dynamic Database Content: Data generated dynamically in response to form submissions (e.g., searching a specific library catalog, internal flight databases, or government record systems directly).

C. Unlinked / Orphaned Pages: Pages without inbound links or explicit sitemaps that web crawlers have no path to discover.

D. Blocked Pages: Pages explicitly marked with noindex or blocked from the Googlebot crawler via server directives.

E. Ephemeral & Real-Time Data: Real-time messaging (Slack, Discord, private WhatsApp threads, live social streams) that isn't publicly archived.

F. The Dark Web: All of those non-standard networks requiring specific routing protocols (e.g., the .onion domains).  There are other ways to find this information that you can figure out.  

SearchResearch Lesson: 

What all of this means is that user mental models for online search (or for any AI system!) need to include what’s being operated on IN ADDITION to just a model of how the system functions.  

To be an effective and savvy searcher, you need to know not just about how the tools operate, but also the nature of the content being indexed.  

You need to know, for instance, that you can search for an image by using Google Lens, but that Google Lens will not find your image in a single frame of a video (even though the video is composed of thousands of image frames arranged in a long linear list)!  

But you knew that, didn't you?  

Keep updating your mental model and... 

Keep searching.  

 

SearchResearch (7/29/26): Mental models: Why you have them… and why they sometimes mislead you…

A mental model...

Hamster-powered search (P/C Gemini)

.. is just the collection of what you know about some thing: what that thing is, the way it looks, how to use it.  Just as importantly, a mental model also tells you (a) what a thing normally looks like, (b) its normal functioning, and (c) how to fix it when things go wrong.  

That’s usually the way people define MENTAL MODEL. But wait… there’s more!  

You should know that the mental model doesn’t have to be accurate, in fact, they rarely are.  Few of us have a deep and accurate mental model of the way our car works, yet we use them all the time.  (Can you really tell me how antilock brakes work?)  

For most things, you don’t need a completely correct model—you just need to know enough to know what it is, how to work it, how it behaves, and how fix things when the process goes awry.  

A mental model is whatever a person believes to be true about that thing--it's appearance, the way it acts as it operates, even how long it takes to do its normal behavior.  

Does your fancy AI thing all run on top of a stack of hamsters?  Sure.  Whatever. As long as you can use it and as long as your mental model makes accurate predictions.  (See image above, and remember that famous April Fool's Day joke about Google's Pigeon Rank. I'm sure there are people who believe this is true.)  

Explaining how something works in detail is often very, very hard and to use your thing, people usually don’t need all of those details. Almost nobody can explain how electricity actually works, yet we all use it successfully every day.  The trick is to create enough of a mental model to be useful.  

A mental model guides you in your use of a tool or system. The mental model tells you “If you do X, then Y will happen.”  If you turn on a light switch, the light bulb goes on.  And if the bulb doesn’t go on, the mental model will give you some ideas about what to do next to fix it.  

Having said that, people can have very different mental models of online search… and in the age of AI, it matters if your mental model is correct enough to be useful.  The reason to have a mental model is to be able to use a system AND make simple predictions about what the system will do.  

In other words, I want to talk about mental models for information seeking, and along the way, discuss mental models for information and sensemaking.   

Let’s start by talking about mental models people have for online search systems.  

Over my years of doing research at Google, I’ve seen a lot of interesting mental models. Here are the three most common mental models that I've seen out there.  

1. Database: when you issue a query, you get back everything that it available on the internet that’s on the topic of your query. If you think about it for a bit, one will realize that this can’t possibly be the case—there are too many hits for any search engine to return, and the idea of a “complete set” of results is just dreaming.  By design, a database WILL return everything that matches the query.  They’re superficially similar, but the model falls apart if you test it carefully. 

2. File search: some people have used a file search tool like Apple’s Spotlight or Windows’ Start Menu search tool. They both search the file system for matches in filenames or file contents. But these tools search a slowly-changing set of local content. Internet search systems do their best to index a vast, seething mass of constantly churning content.  

3. Library index: people who grew up in libraries love to think of internet search as a kind of library catalog. These folks believe that everything is described somewhere on a small card by a talented librarian who lists the title, the date, the author, and the major topic areas of the document. Wikipedia is great, with their articles on nearly every possible topic, but that’s not really complete in any sense.  (It’s fairly easy to find documents on the internet that don’t have a title or even a clear topical focus.)

The thing about these kinds of mental models of internet search is that people have formed expectations for what the system is doing when they use it.  

People with a database-model expect complete results and are surprised when they don’t get it.  And since most real databases have very well-defined operators like NOT, these folks are surprised when a search result seems to defy the query.  

If you’re in the business of creating new applications (including AI-based apps), you—the designer or system builder—have to know what MENTAL MODELs your users have of the thing you’re building. And if it’s a totally new kind of thing, realize that people are going to use the closest MENTAL MODEL they can think of and make predictions with that.  

So, one of the most important things you have to do as a designer is to communicate a good mental model.  

The mental model should be clear about what your thing is (and what it isn’t), as well as set strong expectations about what normal operation is, and what to do when that operation goes off track.  

We’ve written before about the lack of understanding of Control-F (i.e., find in text, the most important skill).  The root cause of the problem is that people’s mental model of a web page does NOT include text-findability as a legitimate operation.  

(It’s an interesting self-check: think for a second right now.  When you’re looking at a webpage in your browser, what kinds of things CAN you do?  We’re talking about finding text, so you know that.  Can you find a synonym for the target of your search?  Is that even possible?  If so, how? You probably know you can export or save the web page as a file, but can you somehow extract and save all of the images on the page?  What about getting the definition of an unfamiliar word?  Possible?)  

What to do about giving useful mental models to our users?  

UX master Jakob Nielsen wrote about this a while back in his article about mental models. (This is my paraphrase of his advice.)  

  

If your user has a different MENTAL MODEL than you, there are two different options:

1 Make the system align with the users' mental models — assuming most users’ models are similar. Your fancy new document editor better look / operate like other document editors, or you’re fighting an uphill battle.  This is the best approach: if people look for something in the wrong place, then move it to the place where they look for it.  If your system doesn’t offer a function that all of the other systems offer, you’d better have a great reason why not.  

2. Improve users' mental models so that they more accurately reflect your system. You can, for example, explain things more clearly, make labels clearer, and generally offer a great communication system in order to guide users to form the right model. Analogies and metaphors work well here--just be sure you know how your user population understands your metaphorical language.  (Caution: It's easy to get your metaphors wrong. See this paper: "On the problem and promise of metaphor use in science and science communication")  

In my next post, I’ll talk about how people think the underlying search systems are built, and what happens when users don’t understand the data over which the search is run.  It’s more complicated than you might expect.  

Keep searching.  

 

SearchResearch (7/10/26): Capture that content... or lose lots of great stuff

 As you know, I’m a researcher… 

… and in my day-to-day work I spend around 5 - 8 hours / day doing online research.  

As a consequence, I end up reading / scanning / sifting / sorting through a lot of material.  And over the past couple of years, I’ve noticed an interesting shift in my notetaking behavior.  

Why you want to capture content: You see, I used to notice something slide past me, and then jot down a quick reminder–maybe a key phrase or something that would let me get back to the original source material.  I usually didn’t write down the URL because I could always just re-search for the thing and get the latest, most up-to-date version.

But that’s not really true anymore.  There are a couple of reasons why I now take notes about everything I want to recall.  

First, there is an illusion of permanence. 

But... The web is fundamentally ephemeral, even though most users treat it as a stable archive.  It is NOT THAT.  Why?  What goes wrong?

• Link Rot: The sheer decay rate of URLs. Even highly credible sources restructure their sites, drop legacy pages, or go behind paywalls.

• Content Drift: The page might still exist, but the specific paragraph, image, or data point you found has been quietly edited or removed.

• Platform Enclosure: Forums, older platforms, or specific social threads disappear entirely when platforms shut down or change their API access.

And then, the mechanisms we use to find information are not consistent over time, making re-doing that search incredibly difficult.

• Search Ranking Shifts: The query that surfaced that one, perfect, golden link today might bury it on page four next month due to algorithm updates or personalized search histories.

• The AI/LLM Factor: This is critical right now. Re-finding information in the era of generative AI is uniquely challenging. If a user gets a perfect synthesis from an LLM, trying to reproduce that exact output later is nearly impossible because of the non-deterministic nature of the models.

What this means is that relying on search as an "external hard drive" creates cognitive blind spots.

First, there’s the "Google Effect": We have been conditioned to remember how to find information rather than the information itself. When the pathway degrades, the knowledge is lost entirely.

Second, there’s Context and Query Loss: When you try to re-find something six months later, you rarely remember the exact, highly specific query string you used the first time. You also lose the peripheral context—the "trail of breadcrumbs" that led you there.

Capture as Active Sensemaking

Capturing isn't just about taking notes and hoarding data; it is a fundamental step in the research process.

• Friction as a Feature: The act of saving a piece of text, taking a screenshot, or logging a citation forces a moment of active engagement.

• Annotation: A captured piece of information allows you to immediately append your own notes ("Why is this important right now?"). Re-finding strips away this personal context.

Practical Strategies 

• The "Save it Locally" Rule: Try downloading PDFs, use web clippers, or taking scrolling screenshots rather than just bookmarking URLs. (Or, if you live in the cloud, save it to your personal cloud storage. The point is to keep your captures in a stable place.)

• Organizing for Future-You: This is key–you’re building and structuring a personal knowledge management (PKM) system. One of the most important things you can do is to NOT create a digital junk drawer.  I always add a quick note about WHY I’m interested in this thing I just captured.  .

The key to building a successful capture habit is ruthlessly eliminating friction. If saving a piece of information takes more than two seconds or breaks the reader's flow, they won't do it.  BUT.. at the same time, you need a little friction to annotate why you’re capturing this.  

Here are a few ways to capture content easily… 

1. Capture the "Full Context" (web pages)

When the layout, images, and surrounding context matter just as much as the text.

• Print to PDF (Ctrl/Cmd + P): The oldest trick is still one of the most reliable. It freezes the page exactly as it appears, bypasses future paywalls, and creates a universally readable, locally stored file. I do this a lot, always saving the PDF with a file name that tells me what’s salient here. 
  
  

• Use the Wayback Machine Extension ("Save Page Now"): For researchers who need a verifiable, shareable, and citable record of a page before it changes. A single click archives the current state of a public URL to the Internet Archive, generating a permanent link that won't succumb to link rot.
  
  

• Web Clippers (Notion, Evernote, Obsidian): Best for users who already use some kind of Personal Knowledge Management (PKM) system. These browser extensions extract the core text and images, strip away the ads, and dump the content directly into a searchable database. In Chrome I use Web Clipper.  

2. Capturing Screenshots for Data, Quotes, & Charts

I use these methods more than any other, often because I’m on the wrong platform (usually, my phone) specific paragraph, a data visualization, or a fleeting comment in a forum.

• Native OS Screenshots with OCR: Modern operating systems are incredibly good at reading text inside images (Live Text on macOS, Snipping Tool on Windows). Taking a quick localized screenshot (Cmd+Shift+4 on Mac, Win+Shift+S on Windows) is the fastest way to capture a chart or quote. Because the OS indexes the text, the user can search for the words inside the image months later. I use Snagit on my Mac desktop to grab images of things that interest me.

• Drag-and-Drop to the Desktop: Highlighting a block of text on a webpage and dragging it directly to the desktop or a folder instantly creates a .txt snippet file on most operating systems. It’s a zero-click way to grab a quote.

• Screenshots on phones: Learn the screen snapshot shortcut on your phone. For iPhone it’s pressing both top buttons left and right simultaneously; for Android it’s press and hold the Power button and the Volume Down button at the same time. Note that both Android and iPhone let you do this by voice command (“Take a screen snapshot”).  VERY handy.  

3. Capturing the "Un-reproducible" (AI Sessions & Dynamic Content)

With generative AI and highly personalized search algorithms, users must treat their screen as a transient state. You cannot rely on "I'll just ask the AI again later," because the model's non-deterministic nature guarantees a different response.  What’s worse, some platforms (like Facebook or LinkedIn) will sometimes update the screen while you’re doing something else.  You can’t just leave the app and hope that the content will still be there when you come back!  Instead, if you want to capture the whole session, try one of these:  

• Immediate Export: If an LLM gives a perfect synthesis or a highly specific breakdown, use the platform's native export button (e.g., Export to Docs/PDF) immediately. Do not rely on the platform's chat history, which can be wiped, deprecated, or lost if the account is locked.

• Prompt + Response Copying: If exporting isn't an option, users should copy both the prompt they used and the result. The prompt is the intellectual work; the result is the product. Both need to be saved together in a local document.

4. Capturing your Mental Model

Captured information is useless if the user forgets why they saved it.

• The "Forward to Self" Rule: It sounds antiquated, but emailing a link or snippet to yourself with a subject line like RESEARCH: [Topic] - [Why this matters] is very effective. It forces a micro-moment of active sensemaking, and the email inbox acts as a built-in, highly searchable triage system.

• Add a short label or note: Imagine that you see this captured content in a year… will you still be able to reconstruct why you captured this?  (Pro tip: You’ll find that sometimes you’ll look at a note to yourself and say “what??”--learn from that moment.  Think to yourself, “what would have been enough context to remember why I captured this??” 

Treat your browser and research history like a river, not a library. If you see gold in the water, pan it out immediately. If you walk away, the current will wash it downstream and your content will be lost.  

The keys to building a successful capture habit are: 

(1)  to simplify everything. If saving a piece of information takes more than two seconds or breaks your flow, you won't do it. 

And (2) establish a practice of looking through your notes.  This will prevent the “digital junk drawer” from forming.  I have a Sunday morning practice where I rigorously look through all of the week’s notes, deleting everything that doesn’t make sense and incorporating the really valuable stuff into the research topics that I’m working on at the moment.  

Capture it, then harvest it.  Don’t let it accumulate (or you’ll have yet another task that you don’t want to do)!  

Keep searching.

SearchResearch (7/3/26): What you need to know about image search (3/3)

 The most important thing to know about image search... 

P/C Gemini. Prompt:
[usually it works, but sometimes it gets it wrong]

... is that Image Search works pretty well, but is not perfect.  One of the skills you have to develop, as a user of online search tools, is to understand their limits.  When does the AI work well, and when does it fail?  

It's pretty easy to find places where Search-by-image doesn't work.  Many of them involve identifying plants.  

I've written about this before here at SearchResearch. What are those plants? What I said then is still true.  Yes, the AI is improving, but it's still not quite perfect.  

You really need to know this, as it's probably going to be this way for a while.  

Here's today's example.  While walking at the Googleplex, I saw a tree with browning leaves.  Here's the photo I took: 

But when I asked Google Lens what this tree was, here's the answer I got... 

Thing is, I know this particular tree REALLY well--I walk past it nearly every day, spring, summer, fall, and winter. I know for certain that this a California Buckeye (Aesculus californica)--you can get all of the details from the authoritative source, CalFlora.  

But Google search-by-image (aka Google Lens) gets it wrong. I can't really blame it because it looks a lot like a California Sycamore (Plantanus racemosa) in its fall foliage.  But it's only early July, so the California Sycamores look like this: 

A California Sycamore at the beginning of June

What Google Lens doesn't know is that California Buckeyes typically brown up and drop their leaves in mid-summer. A quick search for [when do California Buckeye leaves turn brown] gets you to the University of California Master Gardener's page where you can confirm this.  

Google Lens identification got it wrong. 

Another non-botanical example is this beautifully carved slate plaque that I found as part of an art installation.  

I love the texture and the script, but... I can't quite read it!  Have I had a stroke, or is this in some other language? 

Upload the image to Google looking for an analysis.  What IS that text? Here's what Lens tells me... 

This is interesting, but not particularly helpful. 

A trick worth knowing:  You can scroll up a bit in the search window and--lo & behold--a new query box appears!  It will let you ask an additional question of the search: 

That's great!  I asked the obvious question:  [what language is this text?]   Google replied that it was Russian!  

That's a surprise--it doesn't look like Russian to me, but then I don't know what cursive Russian looks like.  

But I do have a Russian friend who reads and writes with fluency. When I asked, she was astounded that I would ask such a silly question.  "Of course this isn't Russian!"  But she DID know about the plaque, telling me that it was part of an art installation and that she knew who the artist was!  (It sometimes pays to ask humans about tricky questions...)    

Turns out that the artist is Sarah Stiles who makes "Cursive Binary" with her handwriting modified by AI.  

Since this doesn't look like Russian, where did Gemini get the idea that this was Russian?  

Well, that's interesting.  If you look at Stiles website, you'll see that she's of Kalmyk heritage.  (That is, from the Republic of Kalmykia, a federal subject of southwestern Russia situated between the Caspian Sea and the lower Volga.)  

I can't prove it, but I suspect there's a bit of AI leakage from the original query image to the question-answering component.  

SearchResearch Lessons 

1. Search still makes mistakes; AI search does too.  Be sure to check your work. Always.  

2. If you ask the AI to go further, it will.. but that's when you're asking for hallucinations to happen. 

3. You need to know what image identification is good at, and what's not good at.  Plants are a difficult case because they often require very close attention to detail in order to tell one apart from another.  The ID might be close, but not quite right. (And as you know, never use AI to identify mushrooms.  This is another case where the fine details really matter.)  

4. Most search-by-image systems don't let you identify people by name.  Of course, your personal content is different. Both Google Photos and Facebook let you identify people in your collection of pix.  

5. Image recognition is really good at commercial products.  Makes sense, since most of the training material is commercial in nature.  Even partial or incomplete images can often be recognized.  

Keep searching! 

SearchResearch (7/2/26): What you need to know about image search (2/3)

Search by image can literally... 

 

Part of a choir book page from Venice, mid-1300s. P/C Dan. 

... help you understand your world.  Here's a bit of understanding the world that I had just this week.  

This photo shows a piece of a large page taken from a choir book that was taken apart many decades ago.  As a fan of  Early Music, I bought this to capture a bit of the music of the period.  (To hear singing from this kind of score take a listen here.)  

I recognize the 4-line staff (the red lines--modern music notation has 5 lines), and I recognize the Latin text below the notes.  But I'm curious: what is this thing on the far left of the staff?  (The thing that looks like a person with a fat belly, made up from the square notes and a big triangle.)  

The obvious approach is to do a Search-By-Image.  

When I do this, Google tells me that "This image shows stemmed semibreve groups from a 14th-century liturgical manuscript. These musical notations are found on folio 5v of a manuscript in the Bodleian Library." The link to the Bodleian Library takes you to this: 

P/C Bodelian Library
Figure 2. Ave vivens hostia/ Ave vivens hostia/ Organum
(Bodleian Library, lat. liturg. e. 42, fol. 4r).

To understand this, you probably have to go look up what that means. I found that stemmed semibreve groups refer to a specific notation used in 14th-century mensural notation. Instead of representing a single, long whole note, these groups are a series of smaller, subdivided notes (semibreves) strung together and marked by stems and dots to dictate complex rhythms.  

That's great... but it doesn't answer my real question which is what's the big triangle thing?  

I notice that each line of music has a triangle with a square on top, next to a pair of squares that bracket one of the lines (either the 3rd or 4th line).  

I know enough music notation to recognize that the far left edge of the music is where the clef symbol usually goes.  

To find out, I can go into AI mode and ask the question directly about the image: 

This is great and accords with what I already know, but now, of course, let's double check this.  

A quick regular search for [C-clef] leads to several articles about clef notations evolved over the years.  We learn that a C-clef in Gregorian chant notation looks like this (inside the red box):  

That's close, but not quite right.  

Reading further we learn that the F-clef looks like this: 

I also did a search on [mensural notation] and found the Wikipedia article on clefs used in Medieval music notation, with this lovely illustration comparing different clefs, and their evolution over time: 

As you can see, the clef in the original manuscript (at the very top) looks a LOT more like an F clef than a C clef with a decoration.  

In any case, we've resolved the question:  that massive glyph is a clef, showing the singers where the C note was based.  It might be a C clef OR an F clef, but we can leave that to the musicologists to figure out.  We now know what it denotes! 

In a world of music notation that looks like this, with multiple F clefs and a gloriously illuminated intial letter D, it's easy to imagine that the C clef would be transformed to an F clef by the stroke of a monk's pen.  

The "Dragon D" music manuscript. "Deus Omnium" from SC Library.

(Searcher Caution: Bing image search completely messes this up, identifying the original clef image as a piece of Chinese calligraphy.)   

SearchResearch Lesson 

I'll say it again, check your work. Always get a separate source that agrees with your interpretation.  

Keep searching!