Answer (Part 2): What DO we know about history, math, and geography?

Last time we talked about history...  
Now, let's talk about math and geography and how much people know about each.   

In this excerpt from Raphael's fresco The School of Athens, Pythagoras is shown writing in a book as a young man presents him with a tablet.  It shows a diagrammatic representation of a lyre above a drawing of the sacred tetractys.  Do you know Pythagoras and his contributions to mathematics? Do you know why the lyre is significant here? 

1.  Can you find a high-quality (that is, very credible) study of how well the citizens of the United States (or your country, if you're from somewhere else) understand (A) history, (B) geography, (C) mathematics?  

Let's try repeating what we did last time for history.  

     [ American knowledge of math ] 


And we see a similar result: 

Here you go: Lots of results telling us that Americans are terrible at math.  Once again I'll open up the top 10 results in parallel browsing and take a look. 

Even the New York Times has an article with the headline, "Why do Americans Stink at Math?" from 2014.  Although it's a compellingly dismal story about American's ability to do math and why the education system isn't working.  But it refers to the results of studies (but does not give any citation) for their data.  

We must dig deeper, looking at the articles AND who publishes each, AND where they get their data from.  

1. US News and World Report "Why students are bad at math" -- points us to the 2017 National Assessment of Education Progress.  We've seen this data source before in our previous post.  This org is also called NAEP and this report is called "The Nation's Report Card," and summarizes the results of testing across a wide spectrum of US schools for grades 4, 8 and 12.  (I'm alway encouraged by a data source when you can download the data for yourself.  Open data is a sign of a reputable organization, one that's willing to let you look at the raw data source.)  Here's their 2017 Math data set in PDF form.  Here's the top line of that report.  (If you're interested, it's worth looking through the data for all of the metadata about their testing methods, and all of the data exceptions--which all data sets have, but which give me confidence that they took good care collecting this data.)  

Click to see this figure at full-size.  

Summary of this data?  There's been a huge drop in math test scores between 1991 and 2017 almost across the board for grades 4 and 8.  

2.  The Quartz.com article "Americans are spectacularly bad at answering even the most basic math questions." is another dismal headline.  This article points to the PISA studies done by the OECD (Organisation for Economic Co-operation and Development).  As they say on their website, "PISA is the OECD's Programme for International Student Assessment. Every three years it tests 15-year-old students from all over the world in reading, mathematics and science. The tests are designed to gauge how well the students master key subjects in order to be prepared for real-life situations in the adult world."  

This is an interesting comparison source that I hadn't thought about:  How can we measure one country's math understanding?  By comparing test scores with other countries!  

What this this test show?  

"Shanghai-China has the highest scores in mathematics, with a mean score of 613 points – 119 points, or the equivalent of nearly three years of schooling, above the OECD average. Singapore, Hong Kong-China, Chinese Taipei, Korea, Macao-China, Japan, Liechtenstein, Switzerland and the Netherlands, in descending order of their scores, round out the top ten performers in mathematics..."  

Uh oh, this means the US isn't even in the top 10.  Where are we?  You can look at their test data overview here.  And this is the key chart... 

Click to see full size.  

As the overview reports: 

Among the 34 OECD countries, the United States performed below average in mathematics in 2012 and is ranked 27th (this is the best estimate, although the rank could be between 23 and 29 due to sampling and measurement error). Performance in reading and science are both close to the OECD average. The United States ranks 17 in reading, (range of ranks: 14 to 20) and 20 in science (range of ranks: 17 to 25). There has been no significant change in these performances over time.

Meanwhile, mathematics scores for the top-performer, Shanghai-China, indicate a performance that is the equivalent of over two years of formal schooling ahead of those observed in Massachusetts, itself a strong-performing U.S. state. 

Just over one in four U.S. students do not reach the PISA baseline Level 2 of mathematics proficiency – a higher-than-OECD average proportion and one that hasn’t changed since 2003. At the opposite end of the proficiency scale, the U.S. has a below-average share of top performers.... 

3.  The Pew Research Center's report, "U.S. Students' academic achievement still lags that of their peers in many other countries" also points to the OECD / PISA study AND several others, giving a nicely integrated overview of the data.  They put a slightly more optimistic spin on the data.  They tell us that American students' math skills have increased over the past according to the NEAP scores from 1990 - 2015, although there seems to be a small tailing off in 2015... 

Chart from the Pew study. Credit: Pew Research Center.  

They also looked at the PISA data (from above) and show the results slightly differently: 

The US position in world math test scores.  Data from PSA, chart by Pew Reearch.  

I could go on here, but you get the point.  Of the top 10 results on the SERP, 10 had bad news about the state of math education in the US.  Many of the results are from reputable sources, they expose their testing methods, and they share their data sets.  The evidence is pretty overwhelming--the US is not doing a great job teaching mathematics to their students.  There's much to do here in teaching our students how to do math.  


Our other SearchResearch Challenge was about geographic knowledge.  

How is the US doing there? 

Let's use our same approach as before: 

And by doing the same analysis (who wrote the article?  what's their bias?  why did they write this article?)  

The first article is from National Geographic, a well-known (and highly reputable) source of geographic information.  They cite a survey done for them by the Council on Foreign Relations about "What College-Aged Students Know About the World: A Survey on Global Literacy."  The upshot? 

The average score was 55% correct. Just 29% of respondents earned a minimal pass—66 % correct or better. And just over 1 percent—17 out of the 1,203 surveyd—earned an A, 91% or higher. 

Respondents exhibited limited knowledge of issues critical to the United States. Only 28 percent of respondents knew that the United States is bound by treaty to protect Japan if it is attacked. 

This doesn't really surprise me.  I live in a United States that is profoundly inward-looking.  Just out of curiosity I asked [ how many US citizens have a passport ] and found that about 37% of the population has one, compared to Canada’s 60% and the United Kingdom’s 75%. This means that nearly 2 out of 3 Americans can’t even fly to Canada, let alone travel to anywhere else in the world (according to a report from the geography department at UC Santa Barbara).  

But it's distressing.  While doing the research for this article I ran across a 2017 New York Times story, If Americans Can Find North Korea on a Map, They’re More Likely to Prefer Diplomacy, which includes this sobering image.  With North Korea in the news on a daily basis, wouldn't you expect a more accurate hit rate?

Data collected by the New York Times. From "If Americans Can Find North Korea on a Map..."

Out of 1.746 US adults who were asked to click on the location of North Korea (on an unlabelled map), only 36% got it right.  The light blue dots are all of the incorrect locations.  This is crazy.  

This has a real-world consequence.  As the author, Kevin Quealy writes: 

"An experiment led by Kyle Dropp of Morning Consult from April 27-29, conducted at the request of The New York Times, shows that respondents who could correctly identify North Korea tended to view diplomatic and nonmilitary strategies more favorably than those who could not.."

The only factor (e.g. gender, age, education, etc.) that seemed to make much of a difference in locating Korea on a map was "Do you know someone of Korean ancestry?"  

Once again, we have much to do to help our students (and ourselves) understand the world at large.  We live in an international web of countries and businesses--it's useful to at least know where they are!  

Search Lessons 

There's an obvious point here about the remarkable lack of knowledge in mathematics and geography, but that's not the goal of SearchResearch (although I personally feel this is a terrible state of affairs).  

The SRS Lessons are: 

1.  To find reliable data, look for data sets.  If an author isn't showing you the data, be skeptical.  Reliable places tend to link to their open data.  If that's not happening, be skeptical.  

2.  Our query pattern [ American knowledge of X ] seems to work pretty well.  I'd be curious to hear from SRS readers if this works well in other countries.  What did YOU find worked? 

3. Parallel browsing (by opening tabs from the SERP within the window), and then going deep on a topic in a new window, is a remarkably efficient way to do  quick broad-brush research.  




Note:  I'm about to set out on two weeks of travel in a place that might (or might not) have an internet connection.  I'll try to post next week, but if I don't post, don't worry--I'm just having too much fun diving in some exotic corner of the world!  



Search on!