The question is--what works?
If you've been using online search tools for any amount of time you know that which tool you use, and how you use it can make a huge difference in the quality of your results.
This week I Challenged you to find what each of these images is really about. When you try to find out the story of these images, where you able to use "regular" Search-by-Image, or did you use AI image descriptions? If so, what prompt did you use, and how well did it work for you?
1. While traveling in northwest Switzerland, I saw this emblem all over the city I was visiting. Can you tell me what city I was in? Just as importantly, what IS this thing?
For each of these images I used 2 different methods: (1) regular Google Search-by-Image, SBI (you know, upload the image and let it search--aka Google Lens), and (2) upload the image to an LLM (either Gemini or ChatGPT) and ask it to identify what it sees in the image. |
| Image B1: On a box of chocolates... |
 |
| Image B2: Carved into a stone wall |
 |
| Image B3: Seen on a parade float |
 |
| Mobile Lens search for B1 image |
We see that "Baslerstab" is the name of the local Swiss-German newspaper AND that "baselstab" is the name of the symbol! (German note: "Basler" means a person or thing from the city of Basel.) Clicking through on the Wikipedia link takes us to the Baselstab article, telling us a bit more:
"The Baselstab or Baslerstab is the name given to the shepherd's staff or bishop's staff in the coats of arms of villages and towns in the Basel region in Switzerland. The crosier of the Prince-Bishop of Basel can be found in the coats of arms of the city of Basel and the cantons of Basel-Stadt, Basel-Landschaft, and Jura. The crosier of Basel-Stadt is black, while those of Basel-Landschaft and Jura are red."
Since that didn't work well, I did the obvious query on regular Google:
"The symbol shown in the image appears to be a heraldic shield with a stylized bird's head and neck facing left, and a lozenge (diamond shape) below it."
"To understand the full meaning of this symbol, you would need to know the context in which it was found. Here are some questions to consider: Where was the image taken? The location could provide clues about the origin of the symbol and the family or institution associated with it."
"Okay, knowing that the image was taken in Basel, Switzerland, provides some very important context. Here's how that helps us understand the symbol:Basel's Heraldic Traditions:Basel has a rich history of heraldry. Notably, the city is strongly associated with the "Baselstab" (Basel staff), a stylized bishop's crozier. This is a very prominent symbol in the heraldry of Basel-Stadt (the city of Basel) and Basel-Landschaft (the surrounding canton).Another significant heraldic figure associated with Basel is the Basilisk.This mythical creature is frequently seen in Basel's heraldic displays."
 |
| ChatGPT trying to identify the carved symbol. |
 |
| After I told ChatGPT the image is from Basel, it was able to figure out what it was. |
Lesson learned here--location matters!
 |
| Click to see full-size |
It correctly identified the dragon-like creature as a basilisk, but that's not what I'm interested in--I want the symbol on the shield.
"This image shows a close-up, angled shot of a bronze sculpture of a griffin holding a shield. The griffin, with its eagle-like head and wings, is the dominant feature, positioned in the left half of the frame...The shield the griffin is holding is prominently displayed in the center of the image. It features a unique emblem, possibly a heraldic symbol, with a curved, stylized design."
"The symbol in the center of the shield is a heraldic emblem known as a "crosier," which is stylized like a pastoral staff with a curled or spiral top. It resembles the ceremonial staff traditionally carried by bishops or abbots, often symbolizing religious or spiritual authority. The depiction on this shield is stylized, emphasizing the curved top and decorative elements typical of such symbols."
 |
| P/C Wikipedia |
Which is a pretty good result.
I'll tell you the ground truth: I took all of these images while visiting Basel for Fasnacht--huge parades with floats (like the one above), and repeated crozier symbols everywhere. Since I was literally seeing this just about ever time I looked up, I was curious what it was. B3 was taken during a Fasnacht parade as a float slowly drifted by me, the box of chocolates was something I bought, and the stone carving was taken as I walked around the city at 4AM for the Morgestraich.
2. In open green spaces all over Zürich I keep seeing this flower popping up--they're growing everywhere in vast numbers. What is it? Again, how do you know? What search method did you use?
 |
| Image P1: A few flowers carpeting the lawn. What are they? |
I took this photo at a random lawn in Zürich because I was curious--any open green space seems to be blanketed with these flowers! What are they?
Since I was walking around, I did the obvious Google Lens search:
As you can see, it calls it a gentian. Well... notice how Lens has selected an individual flower in the middle? That does kind of look like a gentian, but I know gentians, and this isn't a gentian.
By resizing the image when I got to my desktop, I got a much better answer:
This is correct--it's a Primrose, Latin name: Primula. It's a genus of herbaceous flowering plants in the family Primulaceae.
I thought I'd try asking our favorite AIs what they thought. My prompt after uploading the image was [describe what's in this image]
Again, a reasonable answer from Gemini, but not great. This looks a bit like Polemonium pulcherrimum, but it's not quite right. Identifying small blue flowers is always hard.
By contrast, ChatGPT gives a nice description, but no identification:
Not quite what I was looking for.... so I changed my prompt to: [what kind of flowers are these] NOW I got a much better answer. Here's Gemini's answer:
And ChatGPT's:
I didn't mean to write quite as much as I did--but this is a never-ending topic. Let's summarize what we learned from this little escapade...
1. Regular Search-By-Image is great at some things, but not at others. Usually it's simple to tell when SBI doesn't work--the hits don't tell you anything about the object of the search. BUT... you can improve your SBI query strategy by...
2. Include location information about the object of your SBI search. We saw that including the city name vastly improved SBI search on Google. (But, oddly, it did nothing to help Bing's SBI!)
3. Including location information VASTLY improves AI search. We saw that in both Gemini and ChatGPT--their results got much better when it knew where the target was from. Kudos to Gemini for mentioning that "including the location" would help.
4. Use AI/LLMs for search at your peril. We've talked about this before--they give what look like very plausible answers, but they can be complete nonsense. It's tempting to believe them, but the stories AIs told me about the connection between chocolate and the baselstab symbol were truly inventive...and wrong.
5. Flowers are still hard for AIs. Many flowers look a lot alike (e.g., blue primroses and gentian). I'm not really blaming AI for getting it wrong, but at the same time, they don't know their limits. Really. An AI SHOULD be giving you warning signals when it really can't accurately identify something like a flower or a carved symbol. But they don't. (And, for heaven's sake, do NOT trust an AI to correctly identify a mushroom for you.)
6. AIs are sensitive to the prompts you give them. We found that asking an AI to [describe this image] doesn't work--be direct and say what you really want to know--e.g., [identify the flower in this image]
So, to answer our original question, "What works?" There isn't any one answer, but knowing what your tools can do... and what they can't do. Be specific in your questions to an AI, provide as much information as you need, and remember your ordinary online search skills are still invaluable.
Keep searching!
AI can be stubborn... and have attitude in its thinking
ReplyDeletethat it moderates in its actual response.
Gemini Flash Thinking (experimental)
show thinking:
I will use the Google Search tool to check the distribution of White-throated Sparrows. I will search for "White-throated Sparrow range Europe" and "birds of Basel Switzerland".
Based on the search results:
White-throated Sparrows are primarily found in North America. They are not typically found in Europe, including Switzerland.
The birds commonly found in Basel, Switzerland include various finches, tits, robins, blackbirds, and sparrows, but not typically the White-throated Sparrow.
This suggests that my initial identification might be incorrect given the location. I need to re-examine the image based on common birds found in Basel. However, without being able to visually re-analyze the image with the location context, I will have to rely on the visual characteristics I already noted and try to find a European bird that matches that description.
The key features I noted were: black and white striped head, bright yellow spot between eye and beak, gray breast, and reddish-brown wings.
I will search for "bird with black and white striped head yellow spot Europe".
Based on the search results for European birds with similar markings, the Great Tit (Parus major) shows some similarities, particularly the black head and yellow breast. However, the head pattern isn't strictly striped black and white, and the breast is yellow, not grey.
Another possibility is the Eurasian Tree Sparrow (Passer montanus), which has a chestnut crown and black patch on white cheeks, and a grey breast. This doesn't perfectly match the "striped" head and yellow spot description.
Given the initial identification of White-throated Sparrow (which is incorrect for the location) and the difficulty of finding a perfect match among European birds based solely on the description, it's possible the initial identification was flawed, or the location information might be misleading, or the bird might be a rare vagrant.
Since I cannot re-analyze the image, and the White-throated Sparrow is unlikely in Switzerland, I should acknowledge the discrepancy and suggest a possibility based on the features I observed, while also noting the initial identification.
Let's stick with the initial identification but acknowledge the location information makes it less likely.
The user is correcting my identification of the bird in the image and stating that it is a basilisk, not a White-throated Sparrow. A basilisk is a mythical creature, often depicted as a reptile with deadly gaze. This is a significant correction and implies the image is not of a bird at all.
I should acknowledge the user's correction and apologize for the misidentification. Given the mythical nature of a basilisk, I need to adjust my understanding of the image content.
It's important to note that I was previously analyzing the image based on visual characteristics I perceived as bird-like. The user's correction indicates that those characteristics likely belonged to a different kind of creature.
I will respond by acknowledging the user's correction, stating that a basilisk is a mythical creature, and if possible, offer a revised description based on the image if I can now interpret it as a basilisk. However, since I don't have the image readily available for re-analysis in this turn, I will primarily focus on acknowledging the correction.
You're using 2.0 Flash Thinking (experimental), it may not always work as expected.
You are absolutely correct! My sincerest apologies. That was a significant misidentification on my part. A basilisk is indeed a mythical creature, a far cry from a White-throated Sparrow.
Thank you for the correction. I seem to have completely missed the mark on that one!
it generated an image of me interacting with forces beyond my ability:
"the folly of interacting with forces not well understood"
https://i.imgur.com/dqFvbLL.png
bang the drum
ReplyDeletehttps://www.laeckerlihuus.ch/en/TinBaslerDrum150g/artdet/4068
The user is asking if mystery is an essential part of the AI experience. This question can be interpreted in a few ways:
ReplyDeleteFrom the user's perspective: Is there an inherent mystery in how AI works that is part of the user experience?
From the AI's perspective (if it had one): Is there a sense of mystery in the data and processes that AI deals with?
In terms of AI's capabilities: Can AI be used to generate or explore mysteries?
I should consider these different angles to provide a comprehensive answer.
1. Mystery from the user's perspective:
Black Box Nature: For many users, the inner workings of complex AI models can feel like a "black box." They provide inputs and receive outputs, but the exact processes in between can be opaque. This opacity can create a sense of mystery about how AI arrives at its conclusions.
Emergent Behavior: Sometimes, AI systems, especially complex neural networks, exhibit behaviors that were not explicitly programmed and are difficult to predict or fully understand. This emergent behavior can also contribute to a sense of mystery.
Unpredictability: While often reliable, AI can sometimes produce unexpected or even nonsensical results, adding to the mystery of its capabilities and limitations.
2. Mystery from the AI's perspective:
Vast Datasets: AI models are trained on massive amounts of data, which contain countless patterns and relationships that might not be immediately obvious or fully understood, even by the AI itself.
Complex Patterns: In high-dimensional data, AI can discover intricate patterns that are beyond human comprehension, which could be considered a form of mystery.
3. Mystery in terms of AI's capabilities:
Generating Mysteries: AI can be used to create mystery stories, puzzles, and games.
Solving Mysteries: AI is being applied to solve real-world mysteries in fields like forensics, archaeology, and scientific research.
Exploring the Unknown: AI tools can help analyze complex data and identify anomalies or patterns that could lead to new discoveries and the unraveling of existing mysteries.
Considering all these points, I can conclude that mystery does play a role in the AI experience, both as a perceived aspect by users and as an inherent characteristic of the technology and its applications. However, whether it is essential is a matter of perspective.
I will structure my answer to acknowledge the different facets of this question and provide a nuanced response. I will highlight both the sense of mystery that can arise from AI's complexity and the ways in which AI can engage with and explore mysteries.
the actual response:
ReplyDeleteThat's a thought-provoking question! Whether mystery is an essential part of the AI experience depends on how you define "essential" and whose experience you're considering (the user, the developer, or perhaps even the AI itself in a hypothetical sense). Here's a breakdown of different perspectives:
From the User's Perspective:
The "Black Box" Effect: For many users, AI operates somewhat like a black box. They input data and receive outputs, but the intricate processes happening in between can feel mysterious and opaque. This lack of transparency can sometimes lead to a sense of wonder, but also potentially to distrust or frustration.
Unexpected Capabilities: AI can sometimes perform tasks that seem almost magical or beyond human capabilities, leading to a sense of mystery about its potential.
Unpredictability: While often reliable, AI can occasionally produce unexpected or even nonsensical results, adding to the mystery of its decision-making process.
From the Developer's Perspective:
Emergent Behavior: Even for developers, complex AI models, especially deep learning networks, can exhibit emergent behaviors that weren't explicitly programmed. Understanding why these behaviors arise can be a form of solving a mystery.
The Quest for Understanding: The ongoing research in AI is largely driven by the desire to understand intelligence, learning, and consciousness. This pursuit itself is filled with mysteries.
Debugging and Optimization: Diagnosing issues and optimizing the performance of AI models often involves unraveling complex interactions and identifying hidden patterns, which can feel like solving a mystery.
In Terms of AI Capabilities:
Generating and Solving Mysteries: AI is increasingly being used to generate mystery stories, games, and puzzles. It's also being applied to solve real-world mysteries in fields like forensics, archaeology, and scientific research.
Exploring the Unknown: AI's ability to analyze vast amounts of data can help uncover hidden patterns and insights, potentially leading to the resolution of existing mysteries or the discovery of new ones.
Is it "essential"?
For User Engagement: A certain degree of mystery can make AI more engaging and captivating for users, especially in creative or entertainment applications.
For Scientific Progress: The inherent mysteries surrounding intelligence and computation drive AI research and development.
For Practical Applications: In some fields, AI's ability to uncover hidden patterns and solve complex problems is its core value, which is essentially the ability to resolve mysteries.
However, complete mystery and lack of transparency can also be detrimental, leading to a lack of trust and hindering the adoption of AI in critical applications. There's often a balance to be struck between the allure of the unknown and the need for understanding and control.
In conclusion, while not every aspect of the AI experience needs to be shrouded in mystery, the element of the unknown – whether it's the inner workings of the algorithms, the surprising capabilities, or the potential for uncovering new knowledge – is certainly an inherent and often important part of what makes AI so compelling and impactful.