Answer: What building is this?

There's an important cautionary tale this week.  AIs sometimes get it wrong... especially with image search...

A building in downtown Denver. P/C Dan Russell 

Last week we asked a relatively simple question:  What building is this?  (See above.) 

If you do a regular old Google Image Search, you get the right answer:  This is the El Jebel Shrine, aka the Sherman Event Center in Denver, Colorado. That’s great, and exactly what you’d expect.  Easy peasy.

If you search on Bing Image Search, you also get the right answer: 

But I wouldn’t write an SRS post about something so obvious. 

What IS surprising is what happens when you ask your favorite AI / LLMs about this image.  

Mostly, they get it very, very wrong.

Oddly, after getting it right with Bing Image Search, Microsoft's Copilot gives a terrible answer.  (Once again, the left hand does not know what the right hand is doing.)  

 

This is NOT the Denver Athletic Club.  (It is also made of brick and has arches, but no domes or minaret-like towers.)  

But if you use the visual description ability of ChatGPT, you get another very wrong answer (mostly because it’s trying to find buildings near my Palo Alto, California location–an assumption that seems really bad… especially since none of the buildings it suggests look anything like the image I asked about! 

I thought that maybe I should give ChatGPT a hint, telling it that the building was in Denver. But that didn't work either. In fact, the answers got worse. The buildings it suggested aren't anywhere near the search target!

Maybe I'm just asking the wrong LLM?

Here's Claude's reply:

Claude gets the style correct, but not a proper identification.

I then asked Google Gemini what the answer might be.  AGAIN:  It too is really wrong: 

I know the Advanced Medicine Center Building in Palo Alto--it looks nothing like this.

Well, I thought, that’s because I’m using Gemini 1.5 Pro.  (As you know, there are multiple Gemini models to choose from...)

But when I switched to Gemini 2.0 Flash Experimental (the latest!), I got an even more wrong answer, even though it has a “might not work as expected” disclaimer.  Indeed!  

The Mosque of Ibn Tulun looks a little like the image with domes and towers, but the color, layout, and materials are all wrong.  

HOWEVER, when I tried Gemini 2.0 Experimental Advanced, I finally got the correct answer. 

Notice that it took 3 different tries with different Gemini models to get to the right answer.  That’s not encouraging.  We know that most people will simply accept the first result and not do any follow-on checking.  

Since I took the photo I’ll tell you: this really IS the Sherman Street Event Center, aka the El Jebel Shrine, aka the Rocky Mountain Consistory, and as the Scottish Rite Temple is a historic building in the North Capitol Hill neighborhood of downtown Denver. 

Here’s a great article about it: https://denverite.com/2024/05/09/el-jebel-shriner-mosque-photos/ 

The Moorish-inspired building was constructed in 1907, as a meeting hall for the El Jabel chapter of the Ancient Arabic Order of the Nobles of the Mystic Shrine (the Shriners). It was never a true mosque in the Islamic sense. In 1924, having outgrown the building, the Shriners sold it to the Scottish Rite Masons, who renamed it. In 1995, the Scottish Rite sold the building to Eulipions, Inc. who converted it into a catering and events facility, and it’s been bouncing around the Denver real estate market ever since.  (Although it seems to have recently landed a permanent owner.) 

This is a beautiful example of the Moorish Revival Architecture movement of the early 20th century in the US. This stylistic movement is a variation of Islamic architecture that was introduced in the 17th and 18th centuries in Spain. It's characterized by geometric shapes, arches, and decorative elements like arabesques and ceramic tiles.

Bottom line: The AIs are mostly wrong.  Regular search-by-image is much better.  (Oddly, Tineye.com found nothing, not even a near miss!)  

But what’s worse is that they’re CONFIDENTLY wrong.  There’s no hesitation, no questioning of the plausibility of the results.  When Microsoft CoPilot says that this is the “Denver Athletic Club,” it doesn’t say that it “might be” the club, or that “this image looks a great deal like the Athletic Club, but I’m not 100% sure.”  

Which is disappointing.  

Search Research Lessons

1. Don't trust any image identification that the LLMs give you. We've talked about this before, but REALLY... if they can't identify a very visually distinct building, I wouldn't trust their results when foraging for mushrooms, berries, or edible plants! Perhaps one day they'll improve their accuracy... but for the time being, be sure to double-check everything!

As always,

Keep Searching!