Answer: How do you know what to still believe?

What we believe to be true...   

Dall-E. [ digital art young man looking at a skull on the desktop ]

... changes over time. 

That's good because it generally means that we, as a culture, are figuring things out.  It's easy to come up with a list of things that we no longer believe to be true.  

But the bigger Challenge for today is this:  How can we SearchResearchers come to understand how an idea is changing over time?  If we learned something a while ago (say, when you were half the age you are now), should you still believe that this is true?  

In the Challenge last week I gave two examples of psychology discoveries that might (or might not) be thought of as true.   

A.  In 1977, a famous paper by Richard Nisbett and Timothy Wilson ("Telling more than we can know") argued that people cannot introspect about cognitive processes.  

Is that true?  Or, more in keeping with this week's Challenge--does the field of psychology still believe this to be true? 

B.  In 1996, another famous social psychology result is that if you primed people with ideas and words about growing old (such as feeble, worried, Florida, elderly, forgetful) then they will walk more slowly after exposure to such materials.  See: "Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action" by  John A. Bargh, Mark Chen, and Lara Burrows (1996).

Is that true?  Or, more in keeping with this week's Challenge--does the field of psychology still believe this to be true?  

With all that as background, let me pose the Challenge with this caveat: Our Challenge is not whether or not both of these results are true or not, but really what techniques should you know to be able to determine if something is still widely believed or not?  

1. Can you find out how well has these two results have withstood the test of time?  Is it still considered to be true?  Is it something we should be teaching in our psychology courses today?  What SearchResearch process would you use to figure this out?  

First off, here I'm going to talk about ideas in the sciences, and not about political or informal knowledge.  (That's a whole other ballgame, though some of the ideas here will carry over to that kind of investigation.)  

A big issue for much of humanity is that some kinds of knowledge get hardwired into the culture and is difficult to displace with more up-to-date knowledge.  (That's kind of the point of Kuhn's Structure of Scientific Revolutions, new knowledge doesn't just diffuse gently into a culture, but has to go through a kind of revolutionary phase where it's not widely believed, but then a major shift happens.)  

For instance, when I was young, a common treatment for burns in the home was to smear butter all over it.  That was part of "common knowledge," and widely accepted.  (It is NOT the best way to treat a household burn.)  

In the case of social psychology results from above, they both seem to defy "common knowledge."  Nisbett and Wilson's paper tells us that people seem unable to talk about what they're actually thinking--does that sound right to you?  While Bargh's paper tells us that simply priming people with "old words" actually changes their behavior (especially visible in their walking speed).  

To verify the current thinking about these two papers, I'd start by doing a few searches for articles about the original papers and claims: 

A. Search for the claims made in the original papers.   In particular, make sure you’ve got the right search terms (what’s in the popular press may not be what's written about in scholarly articles). Use author names and the terms that they use.  For instance, Nisbett uses phrases like "introspection" and "cognitive processes."  Bargh uses terms like "automaticity" and "stereotype activation."  Start your searches with authors + key phrases that are not likely to have other meanings.  

And, of course, when you find a hit, read it carefully.  Does the newer article support the findings or contradict them?  Does it give new insights or shift in interpretation?  

B. Search for follow-up articles with the "in" search pattern.  In particular, look for ones that use the title of the original paper (or finding) in THEIR title.  For instance, I found this article, What's next for psychology's embattled field of social priming,  by searching for [Bargh "social priming"].  Another method is to find articles by searching for an explicit mention of the first paper  (For example, a search like this, where I'm searching for the lead-in word "in"  [“in Telling more than we can know”] leads to many articles about that original paper.)  

C. Try an LLM – you might get some clues.  My prompt to Bard (which gave a reasonable answer) was: 

[ In Telling More Than We Can Know, Nisbett and Wilson (1977) demonstrated the limits of one's ability to introspect about the nature of cognitive processing. But work has been done since then.  How well has this idea lasted over the past 40 years.  Is this result widely believed now?] 

Bard response: The idea that people are not very good at introspecting about their own cognitive processes has held up well over the past 40 years. There have been many studies since Nisbett and Wilson's (1977) seminal paper that have replicated their findings and extended them to other areas of cognition. For example, one study found that people are not very good at explaining why they made a particular decision, even when they are given all the information that was available to them at the time of the decision (Wilson, Dunn, Kraft, & Lisle, 1989).

This result is widely believed now among cognitive psychologists. It is one of the reasons why so much research in this area is focused on using indirect methods to study cognition, such as reaction time tasks and eye tracking. These methods allow researchers to measure people's cognitive processes without relying on their introspections.

However, it is important to note that the idea that people are not very good at introspecting about their own cognitive processes is not absolute. There are some cases where people are able to provide accurate reports of their thoughts and feelings. For example, people are generally good at reporting the content of their memories (Schacter, 1996).

Overall, the idea that people are not very good at introspecting about their own cognitive processes is a well-supported one. It has been replicated in many studies and is widely believed among cognitive psychologists. However, it is important to note that this is not an absolute rule, and there are some cases where people are able to provide accurate reports of their thoughts and feelings.

It didn't tell me much that I didn't know, but Bard's response DID point me to two additional papers (Wilson et al., 1989; and Schacter, 1996). Unfortunately, the Schacter paper is all about illusory memories, that is, how people THINK they're accurately reporting their memories, but actually are not.  (In other words, Bard has it totally backwards.)  

But despite getting the claim about people being "generally good at reporting the content of their memories," I read Schacter's paper about how people totally do NOT understand the accuracy of their memories... which is exactly what Nisbett was saying.  

My point, though, is that the LLMs can still give you interesting pointers into the literature, even if they don't get the meaning right!  

D. Use Scholar’s “cited-by” feature to see subsequent articles that refer to the original. If I look at the Scholar entry for  “Nisbett and Wilson Revisited” article, I see this: 

Naturally, if you look at the other papers that cite this one, at least a few of them will provide color commentary on how the ideas have withstood the test of time.  

E. Check books as a historical record.  To use Google Books this way, just search for the authors or the key ideas of the papers and see where that takes you.   Here's a search for Nisbett and Wilson.  Here you can see some of the collected opinions of how that original research result from 1977 plays out in current texts.  (Including a book by T. D. Wilson from 2004).  

F. Search for overviews of the field (or authors).  Just searching for [overview <key terms>] often gets you what you need.  Here's that search in Scholar: 

You can get a hint from this that the field of priming research is having a bit of a struggle these days.  It's clear that replicating those studies from the 1990s is turning out to be more difficult than people suspected.  The results were clearly remarkable and changed the field of social psychology, but with subsequent work, it's turning out to be hard to reproduce the effects originally supported.  Upshot: there probably is a real effect of priming, but the behaviors primed are perhaps not as strong and surprising as we once thought.  

As I said, my goal wasn't to teach you about these results in particular, but about how to do follow-ups on work that has been previously published.  In summary: 

SearchResearch Lessons 

A. Search for claims made in the original papers.  Be sure to keep the authors and key concepts in the search.  

B. Search for follow-ups using the "in+title" pattern.  Works more often than you'd expect.  

C. Try an LLM to ask your question.  But as always, verify whatever it tells you.  It might get something totally wrong... 

D. Use Google Scholar's "cited-by" feature. It's a pretty amazing capability to look at other articles that have used those paper in their arguments.  It might be worth checking out a few of those citing papers to see what they say about the original.  

E.  Remember to check books on the topic. It's amazing to me how often I can find a book with commentary on ideas from the past.  

F. Search for overviews of the topic.  Using the simple search pattern [overview + <key terms>] will often get you an article that overviews / summarizes the entire topic.  

Hope you find these search tips useful.  Try them on your next topic!  

Keep searching!