Answer: How good is AI at recognizing images? What should you know?

Search by image is powerful... 

Remarkable desserts. What are they? 

.... but you need to know what it can do (reliably) and what it can't do (unreliably).  

Let's talk about what AI powered image search is capable of doing.  Here are the questions from last week:    

1. The image above (the dessert display) is from a cafe.  Can you figure out what KIND of desserts these are?  Yes, I know you can read the labels, but these are from a particular region of the world.  What kind of cafe is it?  (Image link to full image.)

The obvious thing is to do a Search-By-Image (which we last discussed in January, when searching for the  El Jebel Shrine, aka the Sherman Event Center in Denver.  That was just 11 months ago, but the world has shifted since then.  

We can download the image (with the link above) and do an image search (no longer called "reverse image search" since the function no longer does "reverse" image search, but tries to do an analysis of the image).  You'll get this: 

This is nice, but it's NOT a "reverse image search" in the way we used to think of it.  

To get that function, I'd use Bing image search, which gives you a result like this: 

In this case, there's no exact match for the image, but there are a lot of similar Middle Eastern restaurants and cafes full of yummy pastries.

On the other hand, the Google answer is interesting.  There's a good description of the contents of the pastry case, but over on the right side in the right-hand side panel you'll see a suggested "possibly relevant" link to Sana'a Cafe in Oakland.  

It's a bit of a spoiler, but this IS an image of the pastry case at Sana'a Cafe in Oakland, California!  The big question for us: How does it know?  This is definitely NOT the closest Middle Eastern cafe to my house (which is where I'm writing from).  

I checked to see if it was using the GPS location stored in the photo. 

(Remember that you can pull the lat/long of the image?  Previous SRS discussion about EXIF and the metadata attached to your images.)  

To check, I edited the image metadata to alter the lat/long and re-ran the query--and got the same answer!  

So what IS going on?  Answer: this image has a close-match to an image found in Reddit about the Sana'a Cafe in Oakland!  

Notice that you can get to the "similar images" section  by simply scrolling down the page to get to "Visual matches," where 3 of the top 5 visually similar images are from Sana'a Cafe.    (Note that these images are really similar to the way Image Search used to work--it would show you the nearest matches.)  

That's great--at least we now know how to get the old search behavior to function.  

Back on the first AI-augmented search page, you probably noticed that there's an option to "Show more."  Clicking on this button will give you a more detailed analysis of the image.  It looks like this: 

So.. yeah. Not a lot of help here--this is just a repeat of what we saw in the first frame.  But what happens if you click the "Dive deeper in AI Mode" button? 

Ooops. Now Image Search is going off the rails.  How does Google know that it's the Levant dessert cafe and bakery?  Completely unclear.  And no amount of asking it would give me any useful chain of reasoning.  

Rather than using plain Google Image Search, I thought I'd give Gemini a chance.  One MIGHT hope that the answers would be the same (it's the same company, right?). So I uploaded the image to Gemini and asked it to describe the image.  No surprise, it gave me more-or-less the same answer.  

But when I asked Gemini a follow-up questions [where is this dessert case located] the Google train goes off the rails and into the river where it crashes and burns.  

This is the equally incorrect response, although incorrect with a florid explanation that's completely wrong: 

As much as I admire the idea of reading the reflected text of the logo (which reminds me of what we did in SRS 2012 ("Where are you?"), in this case, it's totally wrong!  I can't see the "Kunafas" anywhere in the image (can you?).  

So I asked Gemini where the "Kunafas" came from.  Here's what I got when I asked: 

Seems good, right?  But let's look at the highlighted region carefully, shall we?  Here, I put the original image and the Gemini-created image side-by-side.  

As you can see, the "reflected letters" are clearly--at least to you and me--the letters of the cafe's name, Sana'a.  The "F A N U K" are all hallucinated.  

Even more bizarrely, I was curious and re-did the original query on regular Google Image Search, using the same image as before and asked Google Image search to describe the image.  This time, it suggested that the place might be the Sana'a Cafe... but again, not reasoning about why.  I assume it's using the "related images" feature and extracting the name from the Reddit thread images.  This is bizarre because it's NOT the same answer from earlier!  

Bottom line: You absolutely have to check everything that Image Search tells you.  Don't just accept it as truth--it could be very far from the truth.  

2. Here's a photo I took while on a walk in San Francisco the other day.  What a strange, strange place!  It's clearly supposed to have a statue on top of the pedestal.  What happened here?  Why is it bereft?  (Image link)  

I did the same process as before:  regular Image Search on Google and get this as an answer: 

The AI overview is completely wrong.  This is NOT at Lands End park at all... everything in this result is wrong.  

On the other hand, the "Visual matches" section actually gives good results.  This IS "Mount Olympus" (the San Francisco version).  

So, let's try again with the fancy Gemini-powered AI image identification process.  What do we get here? 

The first answer ("...likely the Stairs to Mount Olympus Park in San Francisco..") IS correct, while the "another possibility is the One Thousand Steps Beach Access in Santa Barbara" is quite wrong.  

As before, if you ask Gemini directly (by uploading the picture and asking "where is this image"), you get another kind of wrong answer: 

At least it got the trees right (they are Monterey Cypress), but everything else is seriously wrong.  

First off, there IS NO Hilltop Monument at The Sea Ranch.  (I've been there quite a bit, and I'm 99.9% sure such a place doesn't exist.)  Google might mean the Sea Ranch Chapel, but it's not called the Hilltop Monument, and it's not on a hilltop in any case--it's in the flatlands.  

I thought maybe I'd give ChatGPT a chance, but that didn't work either: 

Again with the Lands End?  The only connection is the Lands End also has a lot of Monterey Cypress, but there's no other connection here.  And there IS a monument to the USS San Francisco at Lands End, but again, it has nothing to do with this picture.  Hallucinations abound.  

And, once again, the "Visual Matches" section of the SERP gives you a much better result than the AI parts of the result: 

But you, dear Human, can easily pull the GPS lat/long from the EXIF metadata to find this in Google Maps: 

And then, a regular Google search [ Mount Olympus Park San Francisco ] will teach you that Mount Olympus was a park in more-or-less the center of San Francisco, with a pedestal, atop which stood a dramatic statue, "The Triumph of Light." Mysteriously, the statue (made of bronze and weighing probably 500 pounds) vanished from the pedestal years later and has never been found.  (See the backstory here at FoundSF.org) 

The statue that was there: 

Mount Olympus in SF, with the original statue that mysteriously disappeared sometime after 1955. P/C San Francisco History Center, San Francisco Public Library, via OpenSFHistory.org 

And nobody knows where--or even exactly when--the statue disappeared.  The city took it's collective eye off the ball and it just kind of went-away one day in the mid-1950s.  

Bottom line: Don't trust the AI analysis.  Do the research yourself. 

3. Here's a great picture of a cloud that Regular Reader Ramon sent in for identification.  What's going on here?  (Image link)  

P/C SRS Regular Reader Ramon

A regular Google Image search tells us that this is a fallstreak hole, also known as a "hole punch cloud."  

As you'd expect, I checked this out by doing other searches (e.g., for [fallstreak cloud]) and looking at the collection of remarkable and beautiful photos.  In this search, the AI result and "Visual matches" images are all pretty good.  

And now we know that the fallstreak cloud is caused by supercooled water in the clouds suddenly evaporating or freezing, possibly triggered by passing aircraft passing through the cloud and causing a chain reaction. Such clouds aren't unique to any one geographic area and have been seen in many places.  

Bottom line:  This worked quite well--not a huge surprise as the image is very visually distinct and there are literally thousands of posts with images describing what this is.  

4. This little bridge is in a lovely town somewhere in the world.  Can you figure out where it is, and when it was built?  (Image link)

This is a case when image search works quite well.  Luckily, this is a famous bridge with LOTS of photos taken over the years.  

Yes, it's the Pinard Bridge, located in Semur-en-Auxois. It (and much of the town) date to the 12th century.  But it's really hard to determine when it was first built.  It will probably take some time searching in old French histories to figure out the original date. But since it's in the river valley that historically floods, it's been rebuilt many times.  

Regular Reader Arthur Weiss points out that the city's website of Semur-en-Auxois tells us that  "The Pinard bridge, or Pignard on the Belleforest view, provided access to the Pertuisot mountain pasture. It was destroyed or extensively damaged on several occasions by floods, including those of 1613, 1720, 1765 and 1856."

(I also found this website with the search [ville-semur-en-auxois pont pinard] -- this is one of those cases when searching in the local language really helps.)  

So while the date of first construction was probably in the 12th or 13th century, it's been rebuilt so many times that little of the original bridge is now left in place.  It is, as we would say today, and example of the Ship of Theseus (if Theseus' ship is replaced plank by plank over a long time until all pieces of wood have be replaced by newer wood, is it the same ship?).  

Search Research Lessons

1. Be very, very cautious about AI generated results.  As we saw, the results can be very, very wrong. My advice: Try the AI methods, but double-check everything.  You cannot trust that the answer is correct.  

2. Note that "Visual Matches" section of Image search (often below the fold) has the "old style" most similar images from the web.  That section also often has great clues to the actual thing you seek.  Be sure to check that part of the search results as well.  

Keep searching!