Answer: To search, or to AI? That is the question...

The question is--what works? 

If you've been using online search tools for any amount of time you know that which tool you use, and how you use it can make a huge difference in the quality of your results.  

This week I Challenged you to find what each of these images is really about.   When you try to find out the story of these images, where you able to use "regular" Search-by-Image, or did you use AI image descriptions?  If so, what prompt did you use, and how well did it work for you? 

1. While traveling in northwest Switzerland, I saw this emblem all over the city I was visiting.  Can you tell me what city I was in?  Just as importantly, what IS this thing? 

For each of these images I used 2 different methods: (1) regular Google Search-by-Image, SBI (you know, upload the image and let it search--aka Google Lens), and (2) upload the image to an LLM (either Gemini or ChatGPT) and ask it to identify what it sees in the image. 

Image B1: On a box of chocolates... 

Image B2: Carved into a stone wall

Image B3: Seen on a parade float

Here's what I found: 

B1 (gold symbol on black/white background): This image is very focused on the object of interest and SBI does a much better job.  Here's what the Google Lens (mobile) interface looked like when I asked about this: 

Mobile Lens search for B1 image

It STILL doesn't tell me what it is, but at least we learn yet another potentially interesting word: Baslerstab.  Searching for this leads to: 

We see that "Baslerstab" is the name of the local Swiss-German newspaper AND that "baselstab" is the name of the symbol!  (German note:  "Basler" means a person or thing from the city of Basel.)   Clicking through on the Wikipedia link takes us to the Baselstab article, telling us a bit more: 

"The Baselstab or Baslerstab is the name given to the shepherd's staff or bishop's staff in the coats of arms of villages and towns in the Basel region in Switzerland. The crosier of the Prince-Bishop of Basel can be found in the coats of arms of the city of Basel and the cantons of Basel-Stadt, Basel-Landschaft, and Jura. The crosier of Basel-Stadt is black, while those of Basel-Landschaft and Jura are red."  

Excellent.  Now we know that this symbol is the emblem for Basel and is an abstract version of a bishop's crozier.  

Since I told you that the Baslerstab was on a box of chocolates, I was wondering what the connection was between this symbol and chocolate.  Reasonable question for Gemini, yes? 

Unfortunately, Gemini seems to fall into a hallucinatory rabbit hole with this answer. Despite what Gemini says in its answer, as far as I can tell, there is NO connection between Lindt (a large chocolate company in Zürich) and the actual chocolate company that uses the Baslerstab symbol (which is called Laeckerli-Huus, headquartered in Basel).   Aaarrghhh!  Everything in this answer below about the crozier and Lindt is wrong.  

Since that didn't work well, I did the obvious query on regular Google: 

If you click on that first link you'll see exactly the image of the entire box of chocolate and cookies, a nice example of the baslerstab being used as a design element.  

 

That's MUCH better than Gemini.  Another example of doing a simple Google search that works out much better than asking an AI.  

For the record, ChatGPT didn't know either... and also completely hallucinated a connection between the baslerstab symbol and coat of arms of the canton of Fribourg, Switzerland where the chocolate maker Maison Cailler is based.  Utter balderdash.  

B2 (symbol on wall): This was the symbol carved into a stone wall in the city's center.  Sadly, nothing I could do with SBI would work.  Google failed, Bing's SBI failed, and even focusing in tight on the symbol didn't help.  Search-by-Image just couldn't handle this.  

When I turned to Gemini with the wall-carving image, and asked [ identify this symbol ], Gemini replied: 

"The symbol shown in the image appears to be a heraldic shield with a stylized bird's head and neck facing left, and a lozenge (diamond shape) below it."  

Not quite.  

However, Gemini also said (quite reasonably):  

"To understand the full meaning of this symbol, you would need to know the context in which it was found.  Here are some questions to consider: Where was the image taken? The location could provide clues about the origin of the symbol and the family or institution associated with it."  

To which I answered, [ the image was taken in Basel, Switzerland ] to which Gemini replied: 

"Okay, knowing that the image was taken in Basel, Switzerland, provides some very important context. Here's how that helps us understand the symbol:

Basel's Heraldic Traditions:

Basel has a rich history of heraldry. Notably, the city is strongly associated with the "Baselstab" (Basel staff), a stylized bishop's crozier. This is a very prominent symbol in the heraldry of Basel-Stadt (the city of Basel) and Basel-Landschaft (the surrounding canton).   

Another significant heraldic figure associated with Basel is the Basilisk.

 This mythical creature is frequently seen in Basel's heraldic displays."  

Which tells me that this is a baselstab (the crozier symbol) AND that the City of Basel also uses a basilisk as a heraldic element.  

  

But of course I had to check this as well... luckily, it passes the sniff test.  

When I checked with ChatGPT, it also gave me a poor result: 

ChatGPT trying to identify the carved symbol.

But since Gemini needed to know where the image was from, I also told ChatGPT that this image is from Basel.  Guess what?  A new answer emerged!  

After I told ChatGPT the image is from Basel, it was able to figure out what it was.  

Lesson learned here--location matters!   

So I tried again with Google SBI.  Turns out it DOES matter!  

It makes complete sense--given a random image from somewhere in the world, adding LOCATION information would help any identification process (including humans!) to figure out what a symbol is.  

B3 (dragon with shield): Going through the process again... and once again, SBI didn't work, even when I cropped to just the symbol. Here's the "regular" SBI search:

 

Click to see full-size

It correctly identified the dragon-like creature as a basilisk, but that's not what I'm interested in--I want the symbol on the shield. 

Fortunately, Google Lens now has the ability zoom in on just the key feature by dragging the white rounded angle brackets--in this case, to focus in on the symbol.  When you do that, you'll get an okay search--the first two results are good, but the rest are terribly off-topic! 

The good news is that you learn that it's the city symbol of Basel, but you won't find out WHAT it is!  

If you ask Gemini to [ describe the image ], you'll get a nice description:  

"This image shows a close-up, angled shot of a bronze sculpture of a griffin holding a shield. The griffin, with its eagle-like head and wings, is the dominant feature, positioned in the left half of the frame...The shield the griffin is holding is prominently displayed in the center of the image. It features a unique emblem, possibly a heraldic symbol, with a curved, stylized design."

But no amount of me asking what that symbol is could convince Gemini to give me a real answer.  Not a great result.  Even when I gave it the cropped version of the image (showing just the symbol itself), Gemini couldn't say what it was.  

By contrast, ChatGPT did much better:  When I asked [ what's the symbol in the center of the shield ] ChatGPT responded with: 

"The symbol in the center of the shield is a heraldic emblem known as a "crosier," which is stylized like a pastoral staff with a curled or spiral top. It resembles the ceremonial staff traditionally carried by bishops or abbots, often symbolizing religious or spiritual authority. The depiction on this shield is stylized, emphasizing the curved top and decorative elements typical of such symbols."  

So now we have a term to look up: "crosier."  A quick search takes me to Wikipedia where I learn that the word is usually spelled "crozier" and it's a stylized shepherd's crook, which is ALSO the symbol for the city of Basel, as you can see in the image below:

P/C Wikipedia

But then I took my own advice and added Basel Switzerland to the SBI search, and as expected, it really improved things: 

SO... I took my own advice AGAIN... and re-ran the very first query of Gemini ADDING in the information about [...this image was taken in Basel Switzerland].  Voila!  Here's the answer:  

Which is a pretty good result.  

BUT... you have to know to prompt it with all of contextual information!  

I'll tell you the ground truth:  I took all of these images while visiting Basel for Fasnacht--huge parades with floats (like the one above), and repeated crozier symbols everywhere.  Since I was literally seeing this just about ever time I looked up, I was curious what it was.  B3 was taken during a Fasnacht parade as a float slowly drifted by me, the box of chocolates was something I bought, and the stone carving was taken as I walked around the city at 4AM for the Morgestraich.  

2. In open green spaces all over Zürich I keep seeing this flower popping up--they're growing everywhere in vast numbers.  What is it?  Again, how do you know?  What search method did you use? 

Image P1: A few flowers carpeting the lawn. What are they?

I took this photo at a random lawn in Zürich because I was curious--any open green space seems to be blanketed with these flowers!  What are they?  

Since I was walking around, I did the obvious Google Lens search: 

 

As you can see, it calls it a gentian.  Well... notice how Lens has selected an individual flower in the middle?  That does kind of look like a gentian, but I know gentians, and this isn't a gentian.  

By resizing the image when I got to my desktop, I got a much better answer: 

This is correct--it's a Primrose, Latin name: Primula.  It's a genus of herbaceous flowering plants in the family Primulaceae. 

I thought I'd try asking our favorite AIs what they thought.  My prompt after uploading the image was [describe what's in this image] 

Again, a reasonable answer from Gemini, but not great.  This looks a bit like Polemonium pulcherrimum, but it's not quite right.  Identifying small blue flowers is always hard.  

By contrast, ChatGPT gives a nice description, but no identification: 

Not quite what I was looking for.... so I changed my prompt to:  [what kind of flowers are these] NOW I got a much better answer.  Here's Gemini's answer: 

And ChatGPT's: 

Which is also exactly right.  But, as always... double check.  Never trust an AI's identification of flowers, trees, dogs, or exotic symbols found on the packages of yummy chocolate!  

SearchResearch Lessons 

I didn't mean to write quite as much as I did--but this is a never-ending topic.  Let's summarize what we learned from this little escapade... 

1.  Regular Search-By-Image is great at some things, but not at others. Usually it's simple to tell when SBI doesn't work--the hits don't tell you anything about the object of the search.  BUT... you can improve your SBI query strategy by... 

2. Include location information about the object of your SBI search. We saw that including the city name vastly improved SBI search on Google.  (But, oddly, it did nothing to help Bing's SBI!)  

3. Including location information VASTLY improves AI search.  We saw that in both Gemini and ChatGPT--their results got much better when it knew where the target was from.  Kudos to Gemini for mentioning that "including the location" would help.

4. Use AI/LLMs for search at your peril.   We've talked about this before--they give what look like very plausible answers, but they can be complete nonsense.  It's tempting to believe them, but the stories AIs told me about the connection between chocolate and the baselstab symbol were truly inventive...and wrong.  

5. Flowers are still hard for AIs.  Many flowers look a lot alike (e.g., blue primroses and gentian).  I'm not really blaming AI for getting it wrong, but at the same time, they don't know their limits.  Really.  An AI SHOULD be giving you warning signals when it really can't accurately identify something like a flower or a carved symbol.  But they don't.  (And, for heaven's sake, do NOT trust an AI to correctly identify a mushroom for you.)

6. AIs are sensitive to the prompts you give them.  We found that asking an AI to [describe this image] doesn't work--be direct and say what you really want to know--e.g., [identify the flower in this image]      

So, to answer our original question, "What works?"  There isn't any one answer, but knowing what your tools can do... and what they can't do.  Be specific in your questions to an AI, provide as much information as you need, and remember your ordinary online search skills are still invaluable.  

Keep searching!