Answer: How much does it rain in Northern California?

The quick answer is this:  No.  The rainfall for October 2012 was pretty ordinary—in fact, much less than just 3 years ago, when the rain fell heaviest in October 2009.  (Which goes to show you how imperfect human memory is:  I don’t remember October 2009 as being especially rainy, but when I look back at my journal from that year, it’s clear it rained a LOT.) 

As I showed you in my graphics from yesterday, I really wanted to get a single, simple chart showing the rainfall across the past decade of Octobers.  It’s often important to have a clear goal in mind, especially when you start wading into the sea of data that’s out there. It’s really easy to get sidetracked by all of the other beautiful charts out there and lose track of what you’re seeking.  (Which is why I hand-sketched the chart: To make it clear what my goal was.) 

Like many readers, I knew about a couple of sources off the top of my head.  Wunderground is well-known (and is a composite of professional and amateur weather stations), the data quality is variable, but they often have data where there’s no other source available.  (For instance, there’s a Wunderground reporting station in my neighborhood.  Awfully handy.)  And of course, you could have done the obvious search:  [ rainfall SFO data ]  or if you've done this kind of thing before, try [ precipitation SFO ] 

I also knew that NOAA (the National Oceanographic and Atmosphere Administration—the official government weather data collection group) would have data.  The problem with NOAA is usually digging down deep enough through their reams of data to find the one you want. 

Then, someone happen to mention WeatherSpark.com, which has a wonderful set of weather visualizations and the ability to drill down and select more-or-less what you want. 

So I did what any discerning data junkie would do… I got data from all three so I could compare them.

Data collection:  Getting data from Wunderground is pretty easy.  Click on “Local Weather” and find the “History Data” tab.  That will get you to data for a given date.  Once you find the “Monthly” button, you can find the cumulative precipitation for that location (KSFO).  

If you then notice the format of the URL for the monthly data report:

   http://www.wunderground.com/history/airport/KSFO/2012/10/31/MonthlyHistory.html

You can then change the year to 2011, 2010, etc and collection the data.

Pop that into your favorite spreadsheet, and plot the graph.  (Here's the link to my spreadsheet version of these charts.)  

That took me only a couple of minutes.  Of course, it helped that I knew about Wunderground to start with. 

Comparison with NOAA:  To get “ground truth” data from the authority, I started my search at NOAA.gov – it didn’t take me too long to click through their site (which is really the best way to do it—you have to learn what they call things by reading their pages—they often use very technical terms that you have to learn along the way). 

But it was only a couple more minutes for me to get to the National Climatic Data Center (NCDC) and find that I could ORDER (for delivery via email) the data set for SFO.  I ended up on their data-set order page ( ) and got an email from them a few moments later with a link to the SFO data set for 2000 – 2012! 

I downloaded the data (which has a ton of values and cryptic notation), poured it into my spreadsheet, opened in up and… realized I needed to go read the documentation.  This is professional data, so I really need to understand what things like “Daily HGCN” meant and what that number in the HPCP column meant.  (Turns out they measure rainfall in 1/100ths of an inch—so a 25 in that column is really 0.25 inches.) 

Fine.  I’m a data guy, so I opened up THAT spreadsheet, filtered out all of the non-October values, added up the numbers and got the values for a decade of Octobers.  (The one thing I tripped over was that one of the months had a HUGE rainfall—well over 10,000 inches!  What happened?  Turns out they use a number of 9999 to denote “rainfall not measured,” so I had to go back and clean up the data a bit. Not a big deal, but lesson learned—when there’s something funny in the graph, go check it out.)

Note how similar these graphs are.  This makes me feel good.  Especially that both graphs have 0 inches of rainfall in 2002 and 2003.  

Comparion with WeatherSpark:  Using the WeatherSpark UI (check it out if you’ve never used it) it’s pretty easy to select a decade’s worth of rainfall data from SFO.  I put it into my cart and then went to checkout.  Surprise!  This data costs money!  Given that I already had 2 data sets (including one from the government), I was a little reluctant to spend $15 to get the data, but I figured I’d be willing to do it for the blog. 

So, one credit card transaction later I had the data.  Dropped into my spreadsheet and… guess what… it’s the same. 

Not too surprising I guess, but it’s another lesson learned.   

But the good news is that I feel pretty confident that these graphs represent what has happened over the past ten years.  Including telling me that this past October was just ordinary. 

Lessons learned:  There were a lot to learn here.

1.  Triangulation is a best practice.  Get your data from multiple places.  In this case, WeatherSpark just replicated the NOAA data, BUT they also cleaned it up.  In this case, it wasn’t a big deal—but it could have easily been worth the money for their slicing/dicing and cleaning of the data. 

2.  Check your data.  That error code (9999) might have been easy to miss if the numbers had been larger.  Look for obvious errors (in this case, a giant spike in the data), but also just eyeball it. Sometimes you’ll see things that would escape plotting.  (Example:  Suppose they’d used a -1 to indicate an error.  You’d never seen that in the plot because it would look just like a 0 at these scales, but you could visually pick it out of the data.) 

3.  Some sources turn out to be less useful.  Several readers suggested sites like Wolfram Alpha and ClimateStations.com Both are great for what they do, but I was not able to download the data from either site, which left me unable to do my own analysis.  While their plots are nice, I wanted to do the data checking myself—and for that you need download ability.  (Note that Alpha says they allow downloads, but it’s only of the chart, not the data.)  

Addendum (11/9/12):   

4.  Make sure you're answering the right question.  A few of our loyal readers answered the question... kind-of...  When I was talking with people about this particular problem, it was clear that often they'd slip slightly off the rails and find data about "San Francisco" and never notice that they'd started by searching for data from SFO.  "San Francisco" usually refers to the city (a place that's noted for its very different weather patterns than those at SFO).  (Hat tip to reader Goon for reminding me of this.)

Search on!  (And stay dry…) 

Wednesday search challenge (11/7/12): How much DOES it rain in Northern California?

Sometimes the simplest questions can lead to relatively tricky searches.  This is usually the case when the data you’re looking for isn’t just lying around on the floor of the internet, but is something you might have to assemble yourself. 

I was standing in the microkitchen at work last week waiting to make my café latte when a couple of us started talking about the weather during the past month. 

“Seemed awfully rainy to me,” a friend pointed out.  “Is this something new?  Or has October always been rainy in Northern California?” 

I’ve lived just south of San Francisco for over 25 years, and MY opinion was that this October seemed rainier-than-usual as well.  But you know how impressions are—they’re notoriously subject to all kinds of mis-rememberings, and in particular, the human ability to give accurate summaries of events over the long scale is really quite poor.

So I thought I’d just look it up—but this bit of weather data turned out to be a bit tricky, and leads to this week’s Search Challenge…

Can you find OR create a chart like this one I've sketched below showing each year's total October rainfall at SFO to answer the question:  

Did it rain more in October, 2012 than in any October during the past 10 years? 

My ideal chart for October rainfall at SFO.  (This is made-up data for the purpose of illustration.)  

To make it simple, let’s check the rainfall during October at SFO.  I know that accurate weather data will be available at the airport.  (Although you need not limit your searches to only FAA data. Any reasonably reliable rainfall weather data will do.  It just has to be measured at SFO.)  

As usual, please let us know HOW and WHERE you found the info, along with an estimate about how long it took you to answer the question (or generate the graph). 

Search on! 

How to use the PowerSearchingWithGoogle content in your classes

Lots of teachers and librarians have been asking:  How can I use the materials from the PowerSearchingWithGoogle MOOC in my classes? 

The answer is simple:  You're free to use whatever you'd like, whenever you'd like.  You don't need to wait for us to run the class again.  If you want to offer your own "How to search" class or just uses pieces of the PowerSearchingWithGoogle MOOC, you should feel free to do so.  

The class is at:  http://www.powersearchingwithgoogle.com/course  

You can visit any of the lessons and do any of the activities.  The only thing that won't work are the forum discussions.  We only have those live while we're offering the class.  Truth is, they're a really valuable thing to have (they're part of what makes MOOCs so much fun!), but we only turn those on when we can moderate them and help people with questions that come up.  

But if you wanted to offer a class in your school or library, you should feel free to link to any (or all) of the content in the MOOC.  

If you want to just point to a single video from the class, all you have to do is to click on the YouTube link in the bottom right of the frame.  (See below.)  

That will open the video out into its own YouTube window, and you can then give out that link in your materials, or you could even embed it within your own Google Doc or Site.  

Note that we ALSO have the full text and slides for each of the lessons.  Click on the "Text Version" blue button in the upper right of the frame, or to see just the slides, click on the "Slides" button.   (See above.)  

If you feel inspired, let me know what classes you're running with the PowerSearchingWithGoogle materials.  And if you feel doubly inspired to teach a class with this, maybe we could do a special video Hangout as part of your class!  

Search on! 

Presentation on "What does it mean to be literate in the Age of Google?"

I give a fair number of talks throughout the year, but my favorite presentation in 2012 has been my talk about "What does it mean to be literate in the Age of Google?"   

Since a fair number of people have asked me for the slides, I thought I'd post them here.  Link to the PDF version of my slides. (Note that I haven't included the videos... so you'll have to imagine what those pieces are.)  

If you want to use this, please give attribution; but otherwise, feel free.  

________________________________________


Title:  What Does It Mean To Be Literate in the Age of Google?

Abstract:  What does it mean to be literate at a time when you can search over billions of texts in less than 300 milliseconds? Although you might think that “literacy” is one of the great constants that transcends the ages, the skills of a literate person have changed substantially over time as texts and technology allow for new kinds of reading and understanding. Knowing how to read is just the beginning of it—knowing how to frame a question, pose a query, how to interpret the texts that you find, how to organize and use the information you discover, how to understand your metacognition—these are all critical parts of being literate as well. In this talk I review what literacy is today, in the age of Google, and show how some very surprising and unexpected skills will turn out to be critical in the years ahead.

Answer: What beach am I on?

First off, I have to say I was impressed at the number of people who just recognized the scene.  I hadn’t expected that so many people would look at the picture and say “Oh yeah, that’s the beach at Copacabana—I recognize the mountain  on the left and the pattern in the sidewalk.” 

And yes, the mountain on the far left of the photograph is a dead giveaway (it’s Sugarloaf), but only if you’ve seen it before.  But I’m getting ahead of myself. 

Looking through the comments (and there are hundreds…), it’s clear that most solutions came in two ways:

1.  The most common solution was that the searcher immediately recognized my hint “Nearby there’s a very famous statue that’s clad with an exterior of steatite that’s placed in very, very prominent location.” was a reference to the famous statute “Christ the Redeemer” (or Cristo Redentor, in Portuguese).  The statue is prominently placed atop Corcovado (a local mountain top that's a truly impressive location).  

Most people had to look up what steatite was with a quick

[ define steatite ]

 and recognize it as soapstone.  If you get this part, the rest is easy.  A query like:

[ statue covered in soapstone ]

leads directly to the Cristo Redentor statue in Rio, perched on Corcovado.  

 In my case, I went to Google Earth, found the statue, and looked to see what direction the right hand is pointing.   As you can see in Google Earth, it’s almost exactly due east.  Draw a line, and it hits the beach at Ipanema.  

BUT!  If you then check the picture of the Mystery Beach against a Streetview of the beach at Ipanema, you’ll quickly find that this really isn’t it, and it has to be the next beach to the east.  (Why?  Because the beach at Ipanema faces due south.  You can’t see any mountains from Ipanema--you can spot islands, but not mountains on the mainland. In particular, you can’t see that big Sugarloaf mountain.)  Thus, it has to be Copacabana, which is also more-or-less in the direction that the statue is pointing.  

2.  The other main solution path was to notice the pattern of waves in the sidewalk and then do a search for:

 [ wavy pattern sidewalk beach ]  

 This search leads to both Ipanema and Copacabana as possible answers.  (Which surprised the heck out of me!  I didn’t even think about this as a possible solution path.)  Interestingly, if you leave the term “beach” out of the query, the results aren’t nearly as good.  Again, as with solution 1, if you check the Streetview, you’ll note that the beach at Ipanema has a hexagonal pattern in the sidewalk… Copacabana has a cool wave pattern. 

So.. the rightest answer is Copacabana.  But if you thought it was Ipanema (a beach equally famous in song and story, listen to “Garota de Ipanema” --aka-- “The Girl from Ipanema”), I’ll count your answer correct as well.

After all, when I was digging through my files to find this picture, it was labeled “Ipanema-2010.jpg” – so I had it wrong as well! 

Search lesson #1:  Picking up on ALL of the cues in the photo (wavy lines in the sidewalk, the appearance of the mountains on the horizon) is a great way to verify that your guess is correct.  Streetview to the rescue here—we can nearly match the photo with a Streetview image.  It's important to notice that the sidewalk is very different at Ipanema... See the streetview shot of the beach at Ipanema below.  

Search lesson #2:  As reader Drop the Gloves! pointed out, you can right-click on a Google Map to bring up the “What’s here?” menu option.  That’s how he found out that the location was Ipanema.  This is a great tool to know when you’re looking at a map and can’t quite figure out what’s going on.  It doesn’t work everywhere (we don’t know the name of every possible feature on the planet), but it works often enough (and nearly always in cities) to be really useful. 

To everyone who picked up on the wave vs. hexagonal pattern difference in the sidewalks—Excellent job!  You’re all super-searchers in my book!  I hadn’t thought of that as a possible solution path.  Nicely done.