Answer: How literate are people wrt road signs?

 We live in a world full of symbols... 

Many of which are essential for living a safe life--such as the signs you see on the roadside every day.  

But a real question is to what extent people actually understand common signs and symbols they see every day.  

I assume that you understand all of the signs above, but it's not clear that not everyone shares your skill.  

What's the difference between the first symbol in row 2 vs. the third symbol in row 2. Do you know?   (Answer: "curving road" on the left, "slippery road" on the right.)  

So the Challenge this week is not to identify these signs, but to understand the larger question:   

1.  How many of the most common roadside symbols DO people understand?   

We're looking for a study of how many road signs people understand.  How do you approach such a question?  

Added later:  I should have made it clear that I was asking only about road sign understanding in the US.  As several readers pointed out in the thread over on LinkedIn, different countries have VERY different expectations about how much training one needs before even taking a driver's test.  The US seems to have a fairly low bar for passing the drivers' test!

This is a fairly open question, but I know we're looking for a study of "road sign literacy" that would measure how many people recognize (correctly!) a set of common road signs.  

My first query was: 

     [ how well do drivers understand road signs ] 

Giving this as the first result: 

That's really interesting.  But when you click through to the article, (Many Americans Struggle to Identify Road Signs) you'll quickly find that it's a summary of other work that's just published on the Automotive Fleet website.  (A website for car fleet management.) It's not bad as a summary, but I like to read the original work and get more of the context. 

In this case, that web page summarizes a report by MyVision.org on road sign readability, "Read the Road: Sign Struggles and Dangerous Distractions." Their analysis is pretty good and filled with interesting results. But again, MyVision.org is a LASIK eye surgery provider, not a specialist in reading or automobile safety.  

The main claim in this study is that 20% of respondents did NOT know what these signs mean when based on shape alone,  although they understand "merge" and "yield" correctly. For example, when looking at road signs based on their shape and color but without any text, only 73% of Americans could accurately identify a road construction sign. Obvious question: Do you know what sign this is?  

But as I read this article I find myself filled with questions:  Who did this study?  They say that"We surveyed more than 1,060 drivers"--how did they select the drivers?  Who is "we" in that sentence? How was the survey given to them?  Why did they test road sign shape without text? Questions abound!  We must go deeper.  Does this article lead anywhere useful? 

I have to tell you that I failed to figure out who/when/where the original article was published.  It sounds good and look interesting, but there's no there there.  (I even emailed the people listed as the contact point--no replies yet.)   Update: I got an answer back from the original authors who say that the MyVision.org web page is the only available version of their report; no other methodological details are available.

So we have to take those results with a grain of salt.  They might be great, but who knows? 

I shifted my search to a Scholar.Google.com search: 

     [study understanding road signs] 

And quickly found a relevant article "Comprehension and Training of International Road Signs" published in the very reputable Proc. Human Factors and Ergonomics Society (2004).  (Note: you might have to pay for access.) In this study (done in North Carolina), testing of 100 people (60 women, 40 men) on 100 international road signs discovered that understanding varies a lot!  

Examples: different signs and recognition rates--Stop sign (100%), Road works (92%), road narrows (42%), road closed (29%).  

They also tested the participants after a short training on what the signs mean. Good news: training really works, improving recognition rates to nearly 100%.  (That's not surprising, but what they SHOULD have done was to re-test participants after a couple of weeks to see if the training actually lasted for more than a day.)  

With this search query I also found a fascinating document: The Development and Evaluation of Effective Symbol Signs (1982), a truly masterful study produced by the US National Bureau of Standards to identify the issues around making effective signage that people would understand. (For both road and non-road signs.) In one study (dating from the late 1970s) people were asked to determine what  symbol they were looking at:  

“... the authors found that some symbols, such as ‘telephone,’ ‘no smoking,’ and the conventional U.S. ‘exit’ sign, were understood by almost all the subjects tested. Yet other symbols, such as ‘blind alley,’ ‘do not block,’ and ‘break glass,’ were understood by only 20-25 percent of the subjects. Not only were some symbols not understood, several symbols were given a meaning opposite to that which was intended. Thus, ‘no exit’ or ‘blind alley’ was interpreted as ‘exit’ or ‘safe area’ by almost all subjects who gave a definition for this symbol.”

Ooof.  That's consistent with other studies, but is frightening.  Not only are many road signs not understood, by everyone, but people will latch onto the OPPOSITE interpretation of a sigh.  Bad idea.  

Shift in strategy:  Even though I used the word "literacy" and "understanding" in my framing of the challenge, it became clear by reading these articles that the preferred terms are "comprehension" rather than literacy, and "traffic sign" rather than "road sign" or "symbol."  

New query: 

     [traffic sign comprehension] 

And that in turn leads to a few more studies: "Comprehension of traffic signs with symbolic versus text displays" Transportation Research Part F: Traffic Psychology and Behaviour (2013) This study in Israel shows that symbols alone (that is, without text) are understood about 51% of the time, while symbol + text is nearly 100%.

That's better than the North Carolinian results. BUT they also found that interestingly, signs that were misunderstood to mean the opposite of their true meaning (as in the previous study) were generally not less familiar than signs that were answered incorrectly or partially correctly. Thus, familiarity alone was not a guarantee of good comprehension.

I can go on--but after looking at several studies, the overall conclusion is best summarized as "people mostly recognize the common signs, but rare signs are often not understood, or worse, are thought to mean the opposite of their intended meaning!"  

You could design your own study... if you're curious, here's the master list of all US traffic signs.    

A small section of the US DOT master signage list.

SearchResearch Lessons

1. Persistence!  As usual, the quick and obvious search often gives you information that looks good, but usually requires a bit more digging to validate.  Be one of those searchers who persists.  

2. Learn the preferred terms of art as you read.  As you saw, I shifted my terms as I read the papers.  Pay attention as you search to the language you're seeing.  It can inform your later searches with much better terms.  

Keep Searching! 

The basis of reading literacy is glyph recognition. Just sayin'... 

BTW, this is the throne name cartouche of Thutmose III,  the sixth pharaoh of the Eighteenth Dynasty,.  Details here.  (Men kheper Ra, "Lasting is the Manifestation of Ra" as seen in the Abydos Canon no. 70 and  Saqqara Canon no. 8  Cf: Beckerath, “Handbuch der ägyptischen Königsnamen”, 2nd ed. MÄS 49 (1999). 136-137, 6:T1

SearchResearch Challenge (9/25/24): How literate are people wrt road signs?

I've wondered for a while... 

... to what extent people actually understand common signs and symbols they see every day.  You probably understand all of the symbols in the above image, but based on some of the stellar driving skills I see, it's pretty clear that not everyone shares that skill.  What's the difference between the first symbol in row 2 vs. the third symbol in row 2. Do you know?  

That brings up an interesting SearchResearch Challenge for the week.  How literate are we? The goal of a roadside symbol is to be rapidly understood--you see it / you understand it.  It's supposed to be automatic.  But... 

1.  How many of the most common roadside symbols DO people understand?   

Don't just give me your estimate but find where this has actually been studied, and what the results are.  Are you surprised by what you found?  Do you think the study (studies) are valid and sound?  

Does this make you feel better or worse about your fellow drivers? 

As always, tell us HOW you found your results. 

And... 

Keep Searching! 

The basis of reading literacy is glyph recognition. Just sayin'... 

Answer: What is the oldest city in the Americas?

 Simple questions are

sometimes harder... 

Mythical ancient city by Imagen

... than you'd expect.   

This Challenge is like that.  It's a very straight-forward question that might not have the simplest answer.  It's up to you to figure this out! 

1.  What is the oldest city in the Americas? 

You'd think that just asking a search engine or an LLM would give you the answer and you'd be done.  Right?  

Well... if there's any deep lesson from SearchResearch, it's that things are never as simple as you'd expect... there's always something deeper and more interesting behind the question.  

Every question, even something as straight-forward as this, needs a bit of definition help.  The answer--no surprise--is going to depend on how you define "city" and how you define "oldest."  

First things first: How do we define a city?  Is it just population or "level of sophistication" or some combination?  

By the dictionary definition, a city is just "a large number of people who live fairly close together."  True, that, but not particularly precise.  What's "large" and "close together"?  

Remember that in the year 0, Rome was around 1 million people in size, while London was only around 1.4 sq km (0.5 sq mi) and home to less than 5,000 people.  Of course, by the year 537 AD, Rome's population had fallen to around 30,000 souls, while London had risen to around the same number.  

Populations come and go--cities are built, grow, prosper, decline, and sometimes lose everyone becoming less than a hamlet.  

So we have a couple of definitional questions to answer before we get to the key Challenge. 

a. how many people make up a city? 

b. how large an area does a city have to be?  (Or does population density make a city?) 

c. does the length of time a city is occupied make a difference in our question?  

I mention all of these variables because in order to answer the question, we need to pick some values.  (In some sense, it doesn't really matter which values you pick, as long as most people will agree that "this is a city at this time.")  When Rome fell to 30,000 people, was it still a city?  I'd say so, partly because of history, but also because they were fairly densely packed together.  

So, for our purposes here, a city is an assembly of more than 2000 people living in a small area that supports commercial activity, with some kind of government or ceremonial / religious functions.  (We'll ignore continuity for the moment.  If the city lasted for more than 5 years, it is--or was--a city for our discussion.)  

Once you get beyond definitions, you might think you could just ask all of the ChatBot LLMs and Search Engines this question, "What is the oldest city in the Americas?"  

When you do that, this is what you get:  

  

The answers vary based on differing assumptions that each system makes.  

Bing, for example, only shows St. Augustine, FL... but the first organic result points to the Wikipedia article on "List of cities in the Americas by year of foundation" (we'll look at that in a minute).  

Some of the systems expose their assumptions.  Gemini's reply includes the supposition for each: 

"If you consider the oldest continuously inhabited city founded by Europeans, it's Santo Domingo in the Dominican Republic, established in 1496.    If you're looking for the oldest continuously inhabited city in North America founded by Europeans, that would be St. Augustine, Florida, founded in 1565. However, if you're talking about the oldest city in the Americas overall, the answer is Caral, Peru. It's an archaeological site dating back to around 3500 BCE, making it one of the oldest cities in the world."

That's a good response, but it does NOT expose any of the other cities that might qualify.  (And is pretty North American biased as well since it lists Euro-towns before the older places in Peru, but I digress.)  

If you just look at the cities listed here and collect their founding dates (as given by the search/AI systems), you'll have this:

THIS is why doing a comparison table is a great idea--you can see the different options and the different assumptions that were made.  With this table we can look at each of these claims one at a time.  Let's do a little digging for each of these claims.  

Tlapacoya is claimed by Perplexity to date to 7500 BCE.  That's quite a claim.  By doing a bit of Google searching, it's clearly a city by 1500 BCE, but there are some artifacts going waaay back, including some rather controversial ones dated to 25,000 BCE.  (They're so controversial that we're going to ignore them here.) But where did the 7500 BCE claim come from? 

When I pushed Perplexity on this [why did you say that Tlapacoya dates to 7500 BCE] it rapidly backpedaled and claimed--falsely--that "I did not mention that date."  Harrumph.  Yes you did.  I have the screencapture to prove it.  Sigh. So this seems bogus... The actual founding date of Tlapacoya seems to be 1500 BCE.  

Footnote:  I figured out where the claim of 7500 BCE for the start of Tlapacoya comes from... it was scraped from the Wikipedia article, List of Cities in the Americas without careful verification.  Oops!  There is an error on the internet... Perplexity just forgot what it was trained on.  

Aspero was pretty clearly a city (with major buildings, large temples, and agricultural fields) by 3000 BCE.  It was part of the Caral-Supe civilization which goes back even farther. (The Caral-Supe culture seems to date to 5000 BCE, but cities started later and can be reliably dated to ca. 3700 BCE.)  
Huaricanga was also connected with the Caral-Supe culture and dates to 3500 BCE.  There are more major buildings and temples and possibly a connection to Aspero.  

Caral seems to have had several thousand inhabitants starting around 2600 BCE, centrally located to all of the Carl-Supe sites.

HOWEVER... while reading about Caral, I stumbled across a mention of site that was possibly older--a place called Bandurria.  Curious about this place (which wasn't mentioned by any search engine or LLM), I did a bit of searching and found dates for Bandurria that are around 3000 BCE--older than Caral, but newer than Huaricanga. (See Paleodiet in Late Preceramic Peru: Preliminary Isotopic Data From Bandurria)

Odd, isn't it?  A major city that is contender for oldest city in the Americas, and it doesn't show up in any of the search/LLMs.  

There are a couple other sites that are quite old e.g., Puerto Hormiga in Columbia, or Celilo Falls (aka Wyam) in Washington state--but both of these seem to be ephemeral villages or trading locations--they have long histories as temporary settlements, but never quite made it to city status.  

Bottom line:  The "oldest city in the Americas" tag has to go to Aspero (3710 BCE), with Huaricanga (3500 BCE) and Bandurria (3000 BCE) close behind. All of these cities had more than 2,000 inhabitants, lasted for many years, and were centers of commerce and religion.  

And our new table has moved Tlapacoya to the fifth position, and added Badurria into position three.

SearchResearch Lessons

1. Compare and contrast different sources.  You know, we've talked about "second sourcing" your results. In this case, I compared eight different systems (search engines + LLM chatbots).  As you can see in the table above, the answers are VERY different from each other.  In some cases, the results are just plain wrong.  (Interestingly, not because they're hallucinating, but because they trained on data that was incorrect, which then surfaced in their outputs.)  

2. Building a comparison table is handy. Not just because you can then use the table to work through the different results, but also so you can see the huge variety of results.  When the "answers" are this different from each other, you have to be fairly skeptical... which we found was the right thing to be.  

3. Remember that search results might be incomplete!  I found Bandurria because I noticed the unusual name when scanning the results.  Checking into it, I found the 3rd oldest city in the Americas... and a result that NONE of the systems surfaced!  

4. When doing search comparisons like this, make your definitions clear so people will know what you're comparing. 

Keep searching!  

SearchResearch Challenge (9/11/24): What is the oldest city in the Americas?

 Simple questions are
sometimes harder... 

Mythical ancient city by Imagen

... than you'd expect.  We've seen this multiple times in our SRS explorations.  Simple question can lead to fascinating side-excursions into topics that you didn't expect.  

Today's Challenge is like that.  It's a very straight-forward question that might not have the simplest answer.  It's up to you to figure this out! 

1.  What is the oldest city in the Americas? 

Easy, right?  

When you find the answer BE SURE to tell us what you did to find it AND why you believe this is the oldest city in the Americas.  

There's an interesting reason I'm posing the Challenge in this way.  I hope you'll also find out why I'm bothering to ask what seems to be so simple, and so obvious.  

Keep searching!  

Answer: How can you find search phrases beyond your own brain's power to imagine?

 Expanding your thoughts... 

The "Walkie Talkie" building in central London has a giant unanticipated consequence. At certain times of day, the windows reflect the sun's rays to a point creating a hot spot that can damage cars unlucky enough to be parked at the focus.

  ... is a key step for doing SearchResearch research.

1. What can a researcher do to find other words and phrases that would help in doing online searching for such a topic?  Let's consider my topic--unanticipated consequences--how can we find other helpful search terms and phrases to seek out and understand this topic?  Ideas?  

Note that what we're looking for are other ways to say "unanticipated consequences." While getting synonyms for each of these two words isn't a bad idea (unanticipated = unexpected, surprise, etc. while consequences = effects, results, etc.), you'll miss a bunch of equivalent phrases or terms. From my own reading in the area I knew that phrases like "cobra effect" or "perverse outcomes" were equivalent, but my brain doesn't answer questions like "tell me all of the other terms and phrases" for this idea.

You could do the obvious synonym search with a search engine.

The Challenge this week was to imagine that you're doing research on a topic that's big, complicated, and difficult to render in just a few words.  How do you start to search for such a beast? How do you expand your mind (and search behavior) to other phrases, terms, and ideas? That is, you could do this (note that you need to use double quotes, or you'll get synonyms for "unexpected"):

It's interesting that Google is giving me synonyms for "unforeseen consequences."  I guess that's the same as "unanticipated consequences," but it's a bit odd.  

If I try Bing, I get this, which isn't quite as useful: 

EXCEPT that in the lower right corner, there's a contribution from CoPilot (i.e., ChatGPT4) suggesting that I try "boomerang effect," "collateral damage," or "Dutch disease." (I hadn't heard of Dutch disease before, so that was a new one to me--it means an apparent causal relationship between the increase in the development of a specific sector (for example natural resources) and a decline in other sectors (like the manufacturing sector or agriculture).)  

But I thought this might be a great chance to try out LLMs as a way to get suggestions about equivalent phrases--that is, after all, what they're really good at doing.  

So I prompted each of the top 5 LLMs for [give me a list of the top ten terms or words that mean the same thing as "unanticipated consequences"].  Here's the spreadsheet with the results, which looks like this: 

You can see a pretty good suite of options here, including several that I hadn't thought about ("unforeseeable repurcussion" or "collateral impacts").  

But we had a great set of suggestions from SRS Regular Readers: 

remmij: 

type in "unintended" and let the suggestions populate.

Here's what I get when I follow remmij's strategy.  It's not a bad way to start.  

From Arthur Weiss:  

Now we can use ChatGPT and similar tools. So I put in this prompt: [I'm trying to find books on the topic of "unanticipated consequences". What synonyms should i use in searching for this topic?] 

(Arthur generates the same list I show above.)  

You can combine these with broader concepts such as "policy," "technology," "social change," "economics," or "decision-making" to refine your search further.

I thought it odd that "black swan" wasn't listed so I asked again. (It's useful to have some ideas to back up what the AI gives you).

"Black Swan" is indeed a relevant term when exploring the topic of unanticipated consequences, especially in the context of rare, high-impact events that are difficult to predict. Here are some additional related terms that might be useful:  Black Swan events, Chaos theory, Butterfly effect, Emergent phenomena, Systemic risk, Tipping points, Complexity theory, Cascading failures, Disruptive events, Unknown unknowns, Contingency theory, Rare events, Outliers, Wild cards (used in futures studies), High-impact low-probability (HILP) events

UC Librarian Donald Barclay wrote:  

My go-to for this is to turn to the experts and try to figure out what language they use. In the case of unanticipated consequences, I might search the term in PsycInfo ($) to see if psychology has anything to say on the topic. I might also search the phrase in Social Science Abstracts ($) and/or in an engineering database like Compendex’s “Engineering Village.” ($)  If I find that experts in a field use different, possibly more precise, language to describe what I’m researching, I can then search those terms. An obvious example is that non-experts commonly drop the phrase “split personality” to describe a certain kind of mental illness. Psychologists, on the other hand, use phrases like “disassociative identity disorder” to more accurately describe the phenomenon commonly, but incorrectly, known as “split personality.” By searching the term used by the experts, I’m more likely to encounter information created by experts and less likely to encounter information created by someone who has watched too many episodes of Dr. Phil.

Searching “unintended consequences” in PsycInfo got 340 hits, so I will need to refine that search to make it useful. It does happen that, as an unintended consequence, the first hit in PsycInfo told me about a book I do not know but which seems promising, Good Intentions: Max Weber and the Paradox of Unintended Consequences. Looking at the full record for that first hit, I was treated to some PsycInfo subject terms that might lead me in fruitful directions if I combine them with “unintended consequences.” These are: Choice Behavior (major); Intention (major); Social Processes (major); Analysis; Consequence

Pro tip: When you find a promising article in a database, always look at the subject terms that have been assigned to it.

Krossbow:  

Suggested using PowerThesaurus to find wide-ranging synonyms.   (Dan: This is a great tool that I'd totally forgotten about!)   

 

Here's what it gave me:  unintended consequences, unforeseen consequences, unforeseen effects, their indiscriminate effects, uncontrolled effects, accidental consequences, accidental effects, accidental results, inadvertent consequences, inadvertent effects, incidental effects, unanticipated effects, unanticipated outcomes, unanticipated results, undesirable effects, undesirable outcomes, undesirable reactions, undesirable results, undesired effects, undesired outcomes, undesired reactions, undesired results.  

RB: 

Built Wiki-Guided Google Search (a kind of front end to do effective Google searches over the Wikipedia). That site contains: Wiki-Guided Google Search, which searches on Wikipedia articles to find concepts related to the ones you're searching. When RB tried searching for "unintended consequences," he got query word recommendations which were both specific (North American Free Trade Agreement, Eliza Armstrong case) and conceptual (Perverse incentive, Structural functionalism).

The second tool on that site is Clumpy Bounce Topic Search. It determines what Wikipedia categories a relevant page belongs to, determines the most popular pages in that category, and makes the ten most popular pages available. This time when RB searched for unintended consequences, he discovered the category "Consequentialism" and generated a search containing the terms unintended consequences, "Consequentialism", and "Experience machine", which dropped him deep in to the wilds of philosophy search results.

2. Same question, except this time I want to search for books on the topic of unanticipated consequences.  (Yes, I know I can go to Amazon, Barnes and Noble, AbeBooks, Google Books, or the Internet Archive.)  What's the best way to find the top 10 books on the topic? 

Unsurprisingly, many of the same tricks that worked above also work here.

When I did the cross-comparision of LLMs suggestions for books, I found this:

which was a pretty decent selection.

Donald Barclay wrote:

As for the books part of your question, you can always resort to not just looking at reviews, but considering where a book has been reviewed to get some idea of how it is regarded. Has the book been reviewed in important journals for the field it covers? Has it been reviewed in outlets like the NY Review of Books, the NY Times, the Times, etc.? On the other hand, metrics like “New York Times bestseller” are worthless because it’s beyond simple for publishers to manipulate bestseller lists. You can also look up a book title in a source such as Google Scholar to see how many times it has been cited. A lot of citations is not necessarily an endorsement, but at least it shows you how much attention has been paid to the book. Google Scholar will also direct you to book reviews that might not turn up in Amazon.  

remmij also suggested going to YouTube and searching there for our topic. I've done this before, and I'm glad remmij brought this up. The results are surprisingly rich and worth reading:

SearchResearch Lessons

There are so many here... but here's a quick summary:

1. Try synonym search. Seems obvious, but you'll find a few great ideas.

2. Try using a thesaurus. Krossbow recommends PowerThesaurus, and I see why.

3. Use LLMs and ask for new synonymous phrases. It works surprisingly well, especially if you compare and contrast different LLMs. (Check out my spreadsheet.) This is an especially good use for LLMs in SearchResearch tasks.

4. Check out Wiki-Guided Google Search as another way to look at Wikipedia.  

5. Don't forget about searching on YouTube. There's more there than you might expect.    

Keep searching. 

SearchResearch Challenge (8/28/24): How can you find search phrases beyond your own brain's power to imagine?

 A real challenge... 

The "Walkie Talkie" building in central London has a giant unanticipated consequence. At certain times of day, the windows reflect the sun's rays to a point creating a hot spot that can damage cars unlucky enough to be parked at the focus.

  ... for doing online research (what we call SRS) is figuring out what to use for the search terms and phrases.

For most people and for most research questions, figuring out what search terms to use is not a huge problem--in truth, people mostly look up fairly simple things.  (What's the phone number of my local pharmacy?  When is the swimming pool open?  Is there a grocery store near me?)  The vast majority of online searches are like that--straightforward requests for information.  

But then every so often you land in a really difficult research swamp, and it's tough to figure out how other people would write about the very concepts you're interested in learning about. 

As you no doubt know by now, I'm working on a new book about Unanticipated Consequences (see my earlier post about this).  And even though I'm a full couple of years into the project, I find myself STILL trying to figure out how to find books, articles, web pages, podcasts, news stories (etc.) on the topic. 

That's why today's SRS Challenge is all about how to deal with this kind of problem.  

Suppose you're doing research on a topic that's big, complicated, and difficult to render in just a few words.  How do you start to search for such a beast? More particularly: 

1. What can a researcher do to find other words and phrases that would help in doing online searching for such a topic?  Let's consider my topic--unanticipated consequences--how can we find other helpful search terms and phrases to seek out and understand this topic?  Ideas?  

2. Same question, except this time I want to search for books on the topic of unanticipated consequences.  (Yes, I know I can go to Amazon, Barnes and Noble, AbeBooks, Google Books, or the Internet Archive.)  What's the best way to find the top 10 books on the topic? 

Note that we want to learn HOW you'd find such search terms or books.  What process did you follow to come up with search terms that work for your last big, complex, and tricky search?  

Tell us in the comments! 

Keep searching.