Saturday, July 6, 2019

Answer (Part 2): What DO we know about history, math, and geography?

Last time we talked about history...  
Now, let's talk about math and geography and how much people know about each.   


In this excerpt from Raphael's fresco The School of Athens, Pythagoras is shown writing in a book as a young man presents him with a tablet.  It shows a diagrammatic representation of a lyre above a drawing of the sacred tetractys.  Do you know Pythagoras and his contributions to mathematics? Do you know why the lyre is significant here? 

1.  Can you find a high-quality (that is, very credible) study of how well the citizens of the United States (or your country, if you're from somewhere else) understand (A) history, (B) geography, (C) mathematics?  

Let's try repeating what we did last time for history.  

     [ American knowledge of math ] 


And we see a similar result: 


Here you go: Lots of results telling us that Americans are terrible at math.  Once again I'll open up the top 10 results in parallel browsing and take a look. 

Even the New York Times has an article with the headline, "Why do Americans Stink at Math?" from 2014.  Although it's a compellingly dismal story about American's ability to do math and why the education system isn't working.  But it refers to the results of studies (but does not give any citation) for their data.  

We must dig deeper, looking at the articles AND who publishes each, AND where they get their data from.  

1. US News and World Report "Why students are bad at math" -- points us to the 2017 National Assessment of Education Progress.  We've seen this data source before in our previous post.  This org is also called NAEP and this report is called "The Nation's Report Card," and summarizes the results of testing across a wide spectrum of US schools for grades 4, 8 and 12.  (I'm alway encouraged by a data source when you can download the data for yourself.  Open data is a sign of a reputable organization, one that's willing to let you look at the raw data source.)  Here's their 2017 Math data set in PDF form.  Here's the top line of that report.  (If you're interested, it's worth looking through the data for all of the metadata about their testing methods, and all of the data exceptions--which all data sets have, but which give me confidence that they took good care collecting this data.)  

Click to see this figure at full-size.  

Summary of this data?  There's been a huge drop in math test scores between 1991 and 2017 almost across the board for grades 4 and 8.  

2.  The Quartz.com article "Americans are spectacularly bad at answering even the most basic math questions." is another dismal headline.  This article points to the PISA studies done by the OECD (Organisation for Economic Co-operation and Development).  As they say on their website, "PISA is the OECD's Programme for International Student Assessment. Every three years it tests 15-year-old students from all over the world in reading, mathematics and science. The tests are designed to gauge how well the students master key subjects in order to be prepared for real-life situations in the adult world."  

This is an interesting comparison source that I hadn't thought about:  How can we measure one country's math understanding?  By comparing test scores with other countries!  

What this this test show?  

"Shanghai-China has the highest scores in mathematics, with a mean score of 613 points – 119 points, or the equivalent of nearly three years of schooling, above the OECD average. Singapore, Hong Kong-China, Chinese Taipei, Korea, Macao-China, Japan, Liechtenstein, Switzerland and the Netherlands, in descending order of their scores, round out the top ten performers in mathematics..."  

Uh oh, this means the US isn't even in the top 10.  Where are we?  You can look at their test data overview here.  And this is the key chart... 

Click to see full size.  


As the overview reports: 

Among the 34 OECD countries, the United States performed below average in mathematics in 2012 and is ranked 27th (this is the best estimate, although the rank could be between 23 and 29 due to sampling and measurement error). Performance in reading and science are both close to the OECD average. The United States ranks 17 in reading, (range of ranks: 14 to 20) and 20 in science (range of ranks: 17 to 25). There has been no significant change in these performances over time.
Meanwhile, mathematics scores for the top-performer, Shanghai-China, indicate a performance that is the equivalent of over two years of formal schooling ahead of those observed in Massachusetts, itself a strong-performing U.S. state. 
Just over one in four U.S. students do not reach the PISA baseline Level 2 of mathematics proficiency – a higher-than-OECD average proportion and one that hasn’t changed since 2003. At the opposite end of the proficiency scale, the U.S. has a below-average share of top performers.... 


3.  The Pew Research Center's report, "U.S. Students' academic achievement still lags that of their peers in many other countries" also points to the OECD / PISA study AND several others, giving a nicely integrated overview of the data.  They put a slightly more optimistic spin on the data.  They tell us that American students' math skills have increased over the past according to the NEAP scores from 1990 - 2015, although there seems to be a small tailing off in 2015... 

Chart from the Pew study. Credit: Pew Research Center.  

They also looked at the PISA data (from above) and show the results slightly differently: 

The US position in world math test scores.  Data from PSA, chart by Pew Reearch.  



I could go on here, but you get the point.  Of the top 10 results on the SERP, 10 had bad news about the state of math education in the US.  Many of the results are from reputable sources, they expose their testing methods, and they share their data sets.  The evidence is pretty overwhelming--the US is not doing a great job teaching mathematics to their students.  There's much to do here in teaching our students how to do math.  



Our other SearchResearch Challenge was about geographic knowledge.  

How is the US doing there? 

Let's use our same approach as before: 


And by doing the same analysis (who wrote the article?  what's their bias?  why did they write this article?)  

The first article is from National Geographic, a well-known (and highly reputable) source of geographic information.  They cite a survey done for them by the Council on Foreign Relations about "What College-Aged Students Know About the World: A Survey on Global Literacy."  The upshot? 

The average score was 55% correct. Just 29% of respondents earned a minimal pass—66 % correct or better. And just over 1 percent—17 out of the 1,203 surveyd—earned an A, 91% or higher. 
Respondents exhibited limited knowledge of issues critical to the United States. Only 28 percent of respondents knew that the United States is bound by treaty to protect Japan if it is attacked. 

This doesn't really surprise me.  I live in a United States that is profoundly inward-looking.  Just out of curiosity I asked [ how many US citizens have a passport ] and found that about 37% of the population has one, compared to Canada’s 60% and the United Kingdom’s 75%. This means that nearly 2 out of 3 Americans can’t even fly to Canada, let alone travel to anywhere else in the world (according to a report from the geography department at UC Santa Barbara).  

But it's distressing.  While doing the research for this article I ran across a 2017 New York Times story, If Americans Can Find North Korea on a Map, They’re More Likely to Prefer Diplomacy, which includes this sobering image.  With North Korea in the news on a daily basis, wouldn't you expect a more accurate hit rate?

Data collected by the New York Times. From "If Americans Can Find North Korea on a Map..."

Out of 1.746 US adults who were asked to click on the location of North Korea (on an unlabelled map), only 36% got it right.  The light blue dots are all of the incorrect locations.  This is crazy.  

This has a real-world consequence.  As the author, Kevin Quealy writes: 

"An experiment led by Kyle Dropp of Morning Consult from April 27-29, conducted at the request of The New York Times, shows that respondents who could correctly identify North Korea tended to view diplomatic and nonmilitary strategies more favorably than those who could not.."

The only factor (e.g. gender, age, education, etc.) that seemed to make much of a difference in locating Korea on a map was "Do you know someone of Korean ancestry?"  

Once again, we have much to do to help our students (and ourselves) understand the world at large.  We live in an international web of countries and businesses--it's useful to at least know where they are!  


Search Lessons 


There's an obvious point here about the remarkable lack of knowledge in mathematics and geography, but that's not the goal of SearchResearch (although I personally feel this is a terrible state of affairs).  

The SRS Lessons are: 

1.  To find reliable data, look for data sets.  If an author isn't showing you the data, be skeptical.  Reliable places tend to link to their open data.  If that's not happening, be skeptical.  

2.  Our query pattern [ American knowledge of X ] seems to work pretty well.  I'd be curious to hear from SRS readers if this works well in other countries.  What did YOU find worked? 

3. Parallel browsing (by opening tabs from the SERP within the window), and then going deep on a topic in a new window, is a remarkably efficient way to do  quick broad-brush research.  




Note:  I'm about to set out on two weeks of travel in a place that might (or might not) have an internet connection.  I'll try to post next week, but if I don't post, don't worry--I'm just having too much fun diving in some exotic corner of the world!  



Search on! 

10 comments:

  1. Very interesting Challenge and reading your answers, Dr. Russell

    I tried your [ American knowledge of ] changing American for USA and also tried Mexico and Mexican. With my country at the moment not good results. I even tried in Spanish. The results mostly are quiz to know your level. With American, tried searching topics like video games, sports and sexuality. And found interesting links that I am still reading.

    Also, while reading your answer, remembered that we had a test to know the level of our schools. The test was called Prueba Enlace. I read that now it was changed but, the point is that our schools and teaching method need to change, specially in some regions in which many "teachers" are more focused in everything unless their kids and schools.

    ReplyDelete
    Replies
    1. Just to share some links. With [American knowledge video games]

      Video Games Can Help Us Un­der­stand His­tor­ical Change

      Can video games really create new knowledge?

      Out of topic:

      Inside Chichén Itzá - 360 | National Geographic

      Finally, wish you a great vacation, Dr. Russell. Hope you find new sources for Challenges and have lots of fun. Try to be away from Internet connection the most possible.

      And, congratulations to US and female soccer. Today, US Wins Record Fourth World Cup Title. In just 28 years, they have made history.

      Delete
  2. What's really interesting is the shares of top and low achievers (sadly, the chart only shows this for mathematics). Comparing the US to Canada (because the two countries are similar in many aspects), the US has 2x low performers and .5x high performers despite the average scores of 518 vs 481.

    It would be really interesting to see results broken down within countries, the way it's done for some parts of China. The US is a big country, with a lot of variation. Sometimes I see results broken down by states, but that's still very coarse (California has more income variation than Canada, despite similar population numbers); it would be instructive to see, for example, Bay Area + Seattle + Austin + New York compared to Vancouver + Toronto + Montreal. (For example, I found this: https://www.caschooldashboard.org/reports/ca/2018 but it has no context with even other parts of the US, let alone the rest of the world.)

    My guess is that a lot of the US poor scores are due to school funding inequality (compare American property tax-based funding with https://peopleforeducation.ca/public-education-in-ontario/how-education-is-funded/ ). But that's just a guess. The chart of teacher salaries vs maths scores is also instructive.

    ReplyDelete
  3. Within the US, there is little correlation between school funding and performance, either at the county level or over time (i.e., increases in funding do not lead to better performance). And European school systems deliver comparable or better results to the US at a fraction of the cost ($PPP).

    Another problem with these kinds of comparisons is that the US is compared at the national level against many countries that are scarcely bigger than US states. You can find the US state-level data and compare it against, say, European nations, and you’ll find that the pictures aren’t so different.

    Finally, a lot of the information you find via Google on education (or any other topic of public policy) comes from sources with political agendas and objectives. Naturally, teachers unions and universities tend to present data that are favorable to their political interests; if you really want to understand what’s goin on in education (or any other field), you cannot simply pull statistics from Google and take them at face value.

    ReplyDelete
  4. “This doesn't really surprise me. I live in a United States that is profoundly inward-looking. ... But it's distressing.”

    I don’t understand why this should be “distressing”. The US is blessed by a dgeographic location that permits it not to get involved in wars and conflicts around the world. Americans could live their lives in peace, trade with the rest of the world, and otherwise mind their own business. But since WWII, the military-industrial complex has dominated US foreign policy and the US has stuck its nose into almost every conflict, war, genocide, border dispute, ethnic conflict and trade war around the world, under the pretense of trying to help people, but really just to increase the power and wealth of a select group of people. And this has been a bi-partisan effort: recently, Bush and Obama both did it.

    As an immigrant myself, I’m glad that the American people haven’t yet fallen prey to the idea that it is America’s job to run the rest of the world; that Americans are not interested and don’t have to be interested in the geography, culture, or politics of the rest of the world. I neither expect nor want Americans to know much about where I’m from, and I really would prefer to forget about the country I left as well.

    ReplyDelete
  5. “Our query pattern [ American knowledge of X ] seems to work pretty well.”

    It works pretty well by what metric? How do you know that the answers you get from that query are accurate or representative? How do you know that you correctly interpret the meaning of those statistics?

    I think there is a fundamental error in assuming that you can have meaningful insights into social, economic, or political phenomena by searching for a bunch of sources.

    IOW, searching for “how does open heart surgery work” will get you lots of hits that you can read and that make sense to you, but that doesn’t mean that you can then say something meaningful about how to improve open heart surgery procedures.

    ReplyDelete
    Replies
    1. Methinks that your point is a big part of what SRS is trying to teach--just finding a bunch of resources does NOT guarantee highly reliable content. You're correct, but what I'd like people to take away from this Challenge is that you can find the resources using a pattern like this, but it's up to YOU to assess their quality and credibility. Many of the Challenges touch on large, complex, deep topics. I'm not trying to solve the world's education problem, but I AM trying to help people find content that they can use. Having a query pattern that produces a lot of on-topic results is just the beginning. You have to do the thinking and assessment yourself!

      Delete