SearchResearch Challenge (7/26/23): How do you know what to still believe?

Our beliefs change over time.  

Dall-E. [ digital art young man looking at a skull on the desktop ]

Good thing too. 

It was once widely believed that California was an island, that "bad air" caused disease, and that there was an Open Polar Sea at the top of the world hidden behind miles of Arctic ice.  

(The Open Polar Sea was a hypothesized ice-free ocean surrounding the North Pole. This eventually disproven notion was once so widely believed that many exploring expeditions used it as justification for attempts to reach the North Pole by sea or to find a navigable sea route between Europe and the Pacific across the North Pole. It was a classic mistake for the ages.)

Today, we don’t believe that tooth worms cause dental disease, that we only use 10% of our brain, or that bumps on your head are strong indicators of your intelligence or personality.  

That’s obvious in retrospect for “obviously bogus ideas” that people used to believe.

Bogosity only becomes clear with time.  

However...  It’s harder to see how ideas change when they’re closer to our time and less obviously crazy.  For instance, the idea that multitasking lets you be more efficient at getting many things done was widely held to be true.  But our understanding is slowly changing to an attitude that multitasking is often NOT very effective, and is a bad idea in general.  

Our Challenge for today is this:  How can we SearchResearchers come to understand how an idea is changing over time?  

Here are two such examples that are much closer to us in time than phrenology:  

A.  In 1977, a famous paper by Richard Nisbett and Timothy Wilson ("Telling more than we can know") argued that people cannot introspect about cognitive processes.  

That is, "...when people try to report on their cognitive processes, … they do not do so on the basis of any true introspection. Instead, their reports are based on … [pre-existing beliefs] …about the extent to which a particular stimulus is a plausible cause of a given response.”  

That's an amazing claim when you think about it.  It implies that when you talk about why you made a decision (say, to accept a particular job offer, or to bake a particular dish for dinner), you actually don't know why that decision was made.  You can talk about it, and you might feel as though you have a good understanding of why you made that choice, but the reality is that you're actually making up that story. 

Is that true?  Or, more in keeping with this week's Challenge--does the field of psychology still believe this to be true? 

B.  Another famous social psychology result is that if you give people ideas and words about growing old (such as feeble, worried, Florida, elderly, forgetful) then they will walk more slowly after exposure to such materials.  That's another famous paper, "Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action" by  John A. Bargh, Mark Chen, and Lara Burrows (1996).

This kind of experiment is called a "priming" study.  In other words, a person who is primed with words about the elderly will unconsciously walk more slowly immediately after they read the priming material. This seems wild on the surface of it--would just reading some text about older folks change MY behavior?  That's what the paper argues. 

 

 

With all that as background, let me pose the Challenge with this caveat: Our Challenge is not whether or not both of these results are true or not, but for this week… 

1. Can you find out how well has these two results have withstood the test of time?  Is it still considered to be true?  Is it something we should be teaching in our psychology courses today?  What SearchResearch process would you use to figure this out?  

The general idea here is to figure out a way to determine if an idea's acceptability has shifted over time.  We could ask similar questions about all kinds of socially held beliefs.  Is it true that Moore's Law no longer describes the growth of microelectronics?  How has our understanding about the role of dietary cholesterol shifted over time?  

For our SearchResearcher purposes, what advice would you give to someone who is trying to understand the changes in an idea over time?  

This is the kind of research I do all of the time these days.  I'm writing a new book about Unanticipated Consequences (link to the Substack that's tracking what I'm doing on the book).  

Let us know how YOU would do a bit of SearchResearch to find this out.  Imagine that you're telling a junior researcher how to validate a claim that was made a few years ago.  What would you tell them to do? 

Well?  

Keep searching! 

 

Answer: How can we use LLMs to search better?

Magic is, by definition... 

Precision targeting for SearchResearch. P/C by Mikhail Nilov (Pexels link)

... something that you don't understand.  That's why magicians wow us in their presentations. 

I love magic, but I really don't want magical interfaces to magical systems.   

Still, as we saw in our post about Using LLMs to find Amazing Words..., with a little ingenuity, we can do remarkable things.  (Definition of LLM) 

In that post, I illustrate how to use LLMs to find words that end in -core that describe an aesthetic style.  The clever thing that LLMs did in that Challenge was to find related words that do NOT have the -core ending with a similar aesthetic meaning. (Example: "dark academia") 

Our Challenge this week came in two parts:  

1.  Can you find a way to use LLMs (ChatGPT, Bard, Claude, etc.) to answer research questions that would otherwise be difficult to answer?  (As with the Using LLMs to find Amazing Words... example.  If you find such a research task, be sure to let us know what the task is, the LLM you used, and what you did to make it work.)  

As we've seen before, LLMs are currently not great at providing accurate answers.  So we're trying to figure out ways to use LLMs in productive ways.

In last week's commentary post (on Friday), I showed a way to search for keywords and phrases to use for regular Google search.  I've used that method a few times since then, and it's always worked out well.

Short summary:  Don't ask your LLM for specific answers to questions, and REALLY don't ask for citations.  At the moment, LLMs are all too happy to make up fake citations.  

But you SHOULD ask for other terms and phrases you should be searching for in addition to what you've been searching.  The sample prompt pattern I used was: 

     [what are the  N  most common subtopics related to TOPIC?]  

To use this, just replace the highlighted N with a number (typically 5 - 20), and replace the highlighted TOPIC with a short text description of your topic-to-search-for.  

This is a great way to figure out how to expand your range of ideas. 

2.  Here's an example of this difficult to answer "regular search" task: I wanted to make a list of all the SRS Challenges and Answers (the C&A list) since the beginning of this year.  I used an LLM to help me figure out the process.  Can you figure out what I did?  (I'll tell you now that I learned a bunch in doing this, and it only took me about 10 minutes from start-to-finish. I count that as a major win.)  

Big hint from last week was this..    

I broke this task down into 3 steps:

     1. get the list of C&As from the blog into a text file

     2. extract out the Challenges and Answers (getting rid of anything extra) 

     3. then reverse the order of the C&A list

Let me unpack this.  

1. To get a list of ALL the blog posts, I opened the most recent blog post and scrolled to the bottom. (There are other ways to do this.)  It looks like this: 

Then, I just opened all of the twisty triangles for each of the entries for this year to see each of the blog posts.  That listing looks like this: 

Then, I just selected all of that text, hit COPY, and then opened a text editor and paste the text (unformatted) into a .TXT document.  (I used BBedit, which is a robust and useful editor, but you can use whatever editor you'd like.)   

In this pic, I've highlighted a bunch of lines that are neither Challenges or Answers.  We don't want those lines in our final C&A answer. 

Yes, I could manually delete each line, but what if I want to do 1000 lines?  For that process, I asked Bard: 

There are a couple more answers below, but I realized that grep was exactly what I wanted.  It's a command line action that will find lines that match a pattern and extract them.  Perfect!  

Except that there's a small problem--the code snippet shown here doesn't quite work.  It says that I should do: 

      grep -E "Answer|SearchResearch" bbedit.txt 

After playing around for a while, I figured out that the correct expression should be: 

       grep  'Answer\|Challenge' bbedit.txt

The double-quotes don't work, one needs to use single quotes.  And then you need to use the \ character to tell grep that the | character means OR (in classical Boolean logic).  

Then, once you run that, you've got the list of C&As.  Excellent.  But they're in the wrong order! Dang!  

Now I want to reverse the order of the lines in the file.  That's a classic programming problem, but I don't want to fool around--I just want to flip the order so that the last line becomes the first line, etc.  

Back to Bard:  

This is great!  I didn't know about the tac command, so I've learned something new. 

But when I try to do: 

    

      tac bbedit.txt 

my MacOS terminal application says that it's not part of the terminal commands.  (A useful thing to know: the Linux that ships on the MacOS isn't a "full" distribution--a lot of commands, like tac, are missing.) 

Time to turn to regular Google and search for: 

      [ tac in MacOS ] 

which points me to a StackExchange page with an even better answer, one that doesn't require tac: 

So the right next step was to do: 

      tail -r bbedit.txt

 

which then gave me the file listing of SRS posts BACKWARDS.  That is, only Challenges and Answers from Jan 1, 2023 - July 5, 2023.  As you can see, each Challenge is followed by its correct answer:  

SearchResearch Challenge (1/4/23): How can I find latest updates on topics of interest?

Answer: How can I find latest updates on topics of interest?

SearchResearch Challenge (1/18/23): Musicians travels--how did they get 

Answer: Musicians travels--how did they get from A to B?

SearchResearch Challenge (2/8/23): What do you call this thing?

Answer: What do you call this thing?

SearchResearch Challenge (2/22/23): World's largest waterfall?

Answer: World's largest waterfall?

SearchResearch Challenge (3/8/23): What do these everyday symbols mean?

Answer: What do these everyday symbols mean?

SearchResearch Challenge (3/22/23): What do you call the sediment that 

Answer: What do you call the sediment that blocks a river flowing to 

SearchResearch Challenge (4/5/23): What's this architecture all about?

Answer: What's this architecture all about?

SearchResearch Challenge (4/19/23): How well do LLMs answer SRS 

Answer: How well do LLMs answer SRS questions?

SearchResearch Challenge (5/31/23): Did they really burn ancient Roman 

Answer: Did they really burn Roman statues?

SearchResearch Challenge (6/14/23): How to find the best AI-powered 

Answer: How to find the best AI-powered search engine of the moment?

SearchResearch Challenge (6/28/23): How can you find a free audio book?

Answer: How can you find a free audio book?

SearchResearch Lessons

There are lessons here, and surely more to come as we learn more about working with LLMs. 

1. Ask your LLM to help brainstorm ideas for search terms that you might not have thought about.  If you think of the LLM as a reasonably accurate brainstorming partner, you might it generates some good Google you wouldn't have thought about. 

2. Ask your LLM about ways to transform your data.  I've found that it often will suggest things that I once knew, but forgot about (e.g., using the Linux command tail to reverse the order of lines in a file).  Sometime soon I'll write about other ways I've used an LLM to help me clean and restructure data files.  In truth, this is my primary use case for LLMs these days--as a research assistant to fix up and analyze data. 

3. Be aware that the details of what your LLM tells you might need a little tweaking.  In the above example, I had to tweak that grep expression to use single quotes rather than double quotes.  Often what the LLM tells you is in the right ballpark, but not precisely correct.  (Think of it as a slightly unreliable narrator!)  

Keep Searching! 

Comment: How can we use LLMs to search even better?

An interesting question we should always ask ourselves... 

Precision targeting for SearchResearch. P/C by Mikhail Nilov (Pexels link)

If you've got a new tool (LLMs, for example), as a skilled researcher, you want to find new ways to use that tool.  How can you use that new tool effectively?  

That's really what this week's Challenge is about: What new abilities do LLMs give us?    

Here's one interesting approach that I found--using LLMs to expand your range of thinking about a topic.  Here's an example.  

As Regular Readers will have noticed, I'm a fan of tropical fish, especially in their native tropical habitat. 

One thing I've noticed is that fish in the shallow tropics seem incredibly vibrant and colorful.  At the same time, their more northernly cousins are often brown / gray / drab.  Is this true?  (We asked something similar back in 2022, Does animal color and weight vary by latitude?) 

One of the difficulties of searching for answers to questions like this is that it's difficult to find search terms that are on target and useful.  

In researching this topic ("are fish in the topics more colorful than fish in northern latitudes?") any combination of those words finds a LOT of results, most of which you have to sift through for anything of value.  

I turned to ChatGPT for help in this quest by asking it an open-ended question, carefully NOT trying to bias the answer: 

     [ are fish more colorful in the tropics than in northern latitudes ] 

The result was pretty good.  

Yes, fish in tropical regions are generally more colorful than fish found in northern latitudes. This is primarily due to the differences in environmental conditions and ecological factors between these two regions.

In tropical waters, there is typically a greater diversity of fish species, and many of them have evolved vibrant colors as a result of various factors. These factors include the need for camouflage within colorful coral reef habitats, mating displays to attract mates, and species recognition within highly populated and visually complex environments. The warm and stable conditions in tropical regions also contribute to the proliferation of colorful organisms, including fish.

In contrast, fish in northern latitudes often inhabit colder waters and more temperate environments. The reduced diversity of fish species and the presence of fewer colorful habitats, such as coral reefs, can result in less visually striking fish. Furthermore, the shorter summers and colder temperatures in northern latitudes may limit the availability of resources for colorful displays and breeding.

However, it is important to note that there are exceptions to this general trend, as some species found in northern latitudes exhibit vivid colors for various reasons, such as sexual selection, territorial displays, or mimicry. Overall, while tropical regions tend to showcase a greater variety of colorful fish, it is possible to find colorful fish species in both tropical and northern latitudes.

But without references, I'm not sure how to check the accuracy of this result.  

When I asked for citations with the prompt: 

     [ are fish more colorful in the tropics than in northern latitudes? Give citations to the scientific literature to support your claims ]

ChatGPT begged off saying that "As an AI language model, I don't have direct access to scientific literature or the ability to browse the internet." 

I asked Bard the same question and got a mess of an answer.  Ugh.   

Bard:  Yes, fish are more colorful in the tropics than in northern latitudes. This is a well-established finding in the scientific literature.

One study that investigated this phenomenon was published in the journal Nature Ecology & Evolution in 2022. The study, which was led by Dr. Chris Cooney from the University of Sheffield, found that bird species tend to be 30% more colorful towards the equator. The researchers believe that this is due to a number of factors, including the warmer water temperatures in the tropics, which allow for more complex pigments to develop in the fish's skin.

Another study, which was published in the journal Proceedings of the Royal Society B in 2016, found that the same pattern of increased coloration is also seen in other animals, such as reptiles and amphibians. The researchers believe that this is because the warmer temperatures in the tropics allow for more complex visual signals to be transmitted between individuals.

Here are some of the scientific literature that support the claim that fish are more colorful in the tropics than in northern latitudes:

Cooney, C. J., & West, S. A. (2022). The latitudinal gradient in bird coloration is driven by temperature. Nature Ecology & Evolution, 6(4), 697-704. 

Ghalambor, C. K., Seebacher, F., & Martin, A. P. (2016). The latitudinal diversity gradient of animal colour. Proceedings of the Royal Society B: Biological Sciences, 283(1830), 20160425. 

Shawkey, M. D., & Endler, J. A. (2009). The latitudinal gradient in color: Integrating mechanisms and function. Annual Review of Ecology, Evolution, and Systematics, 40, 211-236.

I hope this helps!

Well.. it doesn't really help.  Notice that Bard switched from talking about fish to talking about birds and animals.  That's NOT helpful.  

What's worse, the citations look like they're totally fabricated.  I cannot find those articles anywhere.  The authors are real people, but the purported articles look like word salad of the articles they've actually published.  

Go home Bard, you're drunk.  

However, if you ask Bard or ChatGPT for ideas for search topics and keywords, it works pretty well.  Here's what ChatGPT told me: 

     [what are the 5 most common subtopics related to tropical fish coloration?] 

Answer (bolding by Dan): 

The study of tropical fish coloration encompasses various subtopics. While it's challenging to definitively list the five most common subtopics, here are five frequently explored areas related to tropical fish coloration:

Camouflage and Cryptic Coloration: Many tropical fish species exhibit coloration that helps them blend into their surroundings, such as coral reefs or rocky substrates. The study of how fish use color patterns to hide from predators or ambush prey falls under this subtopic.

Sexual Selection and Mating Displays: Colorful displays play a significant role in attracting mates and signaling reproductive fitness in many fish species. Researchers often investigate the evolution and function of bright coloration and elaborate patterns related to courtship and mate choice.

Warning and Aposematic Coloration: Certain tropical fish species display bold and vibrant colors as a warning to potential predators that they possess toxic or venomous properties. Understanding the signals and mechanisms behind these warning colors is another common subtopic.

Environmental Influences: The environmental factors that influence tropical fish coloration are of interest to researchers. These may include the impact of light availability, water quality, temperature, and habitat characteristics on the development and maintenance of colors in fish.

Color Change and Adaptive Plasticity: Some tropical fish species have the ability to change their coloration, adapting to different environmental conditions or social situations. Studying the mechanisms and functions of color change, such as camouflage, communication, or physiological regulation, is a prominent subtopic within tropical fish coloration research.

It's worth noting that the study of fish coloration is a diverse and dynamic field, and researchers may delve into various other subtopics depending on their specific interests and goals.

And if you add in the idea of variation by latitude, the subtopics get even better: 

The study of fish coloration by latitude encompasses various subtopics. While the specific ranking of the most common subtopics may vary, here are five frequently explored areas related to fish coloration across different latitudes:

Environmental Gradient and Color Variation: Research focuses on understanding how fish coloration varies across latitudinal gradients, considering factors such as temperature, light availability, and habitat characteristics. This subtopic examines how environmental differences influence color patterns and pigmentation in fish populations.

Adaptation to Different Habitats: Fish species in different latitudes often inhabit distinct habitats, such as coral reefs, freshwater streams, or polar regions. Investigating how fish coloration adapts to these specific habitats, including camouflage, disruptive coloration, or habitat-specific visual signals, is a common subtopic.

Seasonal Variation and Color Change: Many fish species exhibit seasonal changes in coloration, especially in higher latitudes where temperature and light conditions vary significantly throughout the year. Research focuses on understanding the mechanisms and functions of seasonal color variation, including breeding displays, temperature regulation, or response to changing environmental cues.

Reproductive Strategies and Mating Displays: Colorful displays and patterns play a vital role in fish reproduction and mate selection. This subtopic explores the evolution, function, and significance of color traits in courtship rituals, mate choice, and sexual selection across different latitudes.

Latitudinal Gradients in Biodiversity and Color Diversity: Research in this subtopic investigates the relationship between fish color diversity and species richness across latitudes. It explores patterns of color variation in diverse ecosystems and seeks to understand the underlying mechanisms driving these patterns.

This prompt trick ("what are the <N> most common subtopics related to <topic>?") works well.  You plug in a number for N and plug in a topic for <topic> and you're good to go.  

Another variant on this prompt is a way to just get suggested search keywords: 

     [ list without description the 5 top keywords for searching the topic tropical fish coloration variation by latitude ]

ChatGPT: 

1. Tropical fish coloration latitude variation

2. Latitudinal gradients in tropical fish color patterns

3. Color diversity in tropical fish across latitudes

4. Environmental effects on tropical fish coloration by latitude

5. Latitude-dependent variation in tropical fish pigmentation

Bard: 

1. tropical fish coloration latitude

2. tropical fish color variation

3. tropical fish coloration latitudinal gradient

4. tropical fish coloration temperature

5. tropical fish coloration evolution

As you can see, these lists of keyword/phrases to use for "regular search" are very different, but both have really great suggestions about ways you can continue your search.  And they teach you great terms to use ("color diversity" or "latitude-dependent" variation.")  

Bottom line: LLMs can be used to generate really great IDEAS and keywords for continuing your search.  

And, as I've said before, don't trust the LLMs to give you a coherent (or accurate) answer to a complex question.  

Idea for this prompt snarfed from the Search Engine Journal article "How to use ChatGPT for Keyword Research."  Thanks to Dan Smullen for the inspiration. 

SearchResearch Challenge (7/12/21): How can we use LLMs to search even better?

I've been skeptical... 

Precision targeting for SearchResearch. P/C by Mikhail Nilov (Pexels link)

... about the search capability of LLMs. But I realize they're here to stay--no amount of witchcraft will make them go away and we can only wait for them to get better.  

In the meantime, how can we use LLMs effectively in our online research?  Can we learn to use these new tools in effective ways?  

In other words, can we get them to do things that are difficult in "regular search"?  

I think the answer is a clear yes, although as I've pointed out, debugging / fact-checking is required.  

Still, as we saw in our post about Using LLMs to find Amazing Words..., with a little ingenuity, we can do remarkable things.  (In that post, I illustrate how to use LLMs that end in -core.)  

Our Challenge this week is in two parts:  

1.  Can you find a way to use LLMs (ChatGPT, Bard, Claude, etc.) to answer research questions that would otherwise be difficult to answer?  (As with the Using LLMs to find Amazing Words... example.  If you find such a research task, be sure to let us know what the task is, the LLM you used, and what you did to make it work.)  

2.  Here's an example of this difficult to answer "regular search" task: I wanted to make a list of all the SRS Challenges and Answers (the C&A list) since the beginning of this year.  I used an LLM to help me figure out the process.  Can you figure out what I did?  (I'll tell you now that I learned a bunch in doing this, and it only took me about 10 minutes from start-to-finish. I count that as a major win.)  

I show the answer below.  Note that this C&A list is sorted by date.  

List of SRS posts, only Challenges and Answers from Jan 1, 2023 - July 5, 2023: 

SearchResearch Challenge (1/4/23): How can I find latest updates on topics of interest?

Answer: How can I find latest updates on topics of interest?

SearchResearch Challenge (1/18/23): Musicians travels--how did they get 

Answer: Musicians travels--how did they get from A to B?

SearchResearch Challenge (2/8/23): What do you call this thing?

Answer: What do you call this thing?

SearchResearch Challenge (2/22/23): World's largest waterfall?

Answer: World's largest waterfall?

SearchResearch Challenge (3/8/23): What do these everyday symbols mean?

Answer: What do these everyday symbols mean?

SearchResearch Challenge (3/22/23): What do you call the sediment that 

Answer: What do you call the sediment that blocks a river flowing to 

SearchResearch Challenge (4/5/23): What's this architecture all about?

Answer: What's this architecture all about?

SearchResearch Challenge (4/19/23): How well do LLMs answer SRS 

Answer: How well do LLMs answer SRS questions?

SearchResearch Challenge (5/31/23): Did they really burn ancient Roman 

Answer: Did they really burn Roman statues?

SearchResearch Challenge (6/14/23): How to find the best AI-powered 

Answer: How to find the best AI-powered search engine of the moment?

SearchResearch Challenge (6/28/23): How can you find a free audio book?

Answer: How can you find a free audio book?

Please post your answers here in the comments section.  Be sure to tell us the steps you used to get your data into this form.  

If you scroll down far enough, I've added a Big Hint.  

Best of luck in your SRS-ing this week.  

Keep Searching! 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. hint below here... 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

 

Big hint:  I used Bard to figure out how to do this.  

I took 3 steps: 

1. get the list of C&As from the blog into a text file; 

2. extract out the Challenges and Answers; 

3. then reverse the order of the C&A list.