SearchResearch (2/4/26): Be careful about using image search--it hallucinates too! (But Google Lens is the best of the bunch)

 This month I'm teaching... 

Sheffield-Sterling-Strathcona Hall at Yale

... at Yale University in New Haven, CT... that paragon of Collegiate Gothic architecture.  

As a consequence, I've been taking a lot of photos of the buildings, the stained glass windows, and all the ornamentation.  Wonderful!  (I've also been taking in as many music performances as I can.  But I digress.) 

Naturally, I've been trying to use Google Lens and Gemini to identify the buildings.  It's a natural thing to do--what IS that building?  What does that particular sculptural ornament mean?  

I've written about Search-By-Image more than a few times (Modifying a reverse image search, Asking questions of images with AI, What's that logo?) 

But I've had some questions in the past: Image identification is great--when it works.  

Now that we're in an ever-improving, lovely age of AI, have things improved much?

Bottom line:  Sadly, Search-by-image (aka "reverse image search") is still sometimes problematic.  Verify everything. Good to know that Google Lens is your best option.

I don't mind when it makes an error--we all do that.  What I DO mind is when it presents results as authoritative, without any expression of doubt. 

Here are a couple of examples of successes: 

On the left is a photo I took of a particularly intriguing fence post sculpture. Here, I used Google Lens by right-clicking on the image in Google Photos.  The answer here, on the right, is correct--this is a fence post in front of the "Book and Snake" secret society tomb at Yale.  (Aside: The secret societies meet in mostly windowless buildings called "tombs." You could look it up.)  

Here's another example of Google Lens working properly (oddly enough, in front of another secret society tomb, "Scroll and Key"):  

I was curious what Gemini would say about this, so I was shocked to learn that Google Lens and Google's Gemini do NOT AGREE on what this is.  Same photo, but a different part of Google has a different opinion of what this is.. 

This result is SO WRONG.

This is so odd because the image has the EXIF metadata--Gemini should know that my photo wasn't at the corner of Trumbull and Temple streets.  Here's a map showing where I took the photo: 

I was standing on the east side of College Street, shooting west towards the Scroll and Key Tomb.

Of course, the Berzelius tomb actually IS at the corner of Trumbull and New Haven streets.  It looks like this: 

As you can see, the two buildings look NOTHING alike--the Scroll and Key tomb has very obvious striped walls.  Something is screwy somewhere.  

On the other hand, I have to say that Google Lens very often DOES get it right--this is especially true when contrasted with Bing's image search.  Bing got none of these right.  (I can't recommend using it for much of anything.)  

Here's another one--an image I took inside of a dining hall at Yale: 

Once again, Google Lens (invoked from Google Photos) identifies this correctly:  this is the Berkeley College dining hall, "The architecture is characteristic of the Collegiate Gothic style prevalent at Yale, featuring high, ornate wooden ceilings and large windows." 

And, once again, Gemini gets it wrong: "This photo was taken in the Great Hall of Hart House at the University of Toronto in Canada."  It even gives a lot of very specific (and wrong!) details: "The large stained-glass window at the end is known as the Great West Window. It features various coats of arms, including those of the University of Toronto and its constituent colleges..."  

For giggles, I tried Bing image search--which believes it's the Oxford dining hall at Emory University.

I can go on, but let me summarize this.  

SearchResearch Summary 

1. Google Lens is the most accurate search-by-image system out there.  But Lens has limits, it won't search for sexually explicit content; violent or gory content; hateful content; or dangerous content.  And does a terrible job with faces, mostly to preserve individual privacy.  (Though it will identify famouse people.)  

2. LLM-based AIs are not a great image analysis tool. It's odd because Gemini could call Google Lens and improve their accuracy, but apparently those two parts of the company don't talk together.  Don't trust those LLM/AI results.  FWIW, no other LLM / AI tool does a decent job either.  ChatGPT and all the others are just as bad.  

3. Bing reverse image search is wrong a lot of the time.  Ermmm... maybe avoid it?  

Keep searching!