Tuesday, November 25, 2014

How much snow was there? Part 2--Mapping it out with choropleths and heatmaps

I thought I'd try something a little more sophisticated this morning with the snowfall data.  

So I took all the data I got from the NOAA site for all weather reporting sites (in their list), extracted just the data for cities in New York, between Nov 2 and Nov 22.  I dumped this into a table like this: 

This is pretty much just copy/pasted from their CSV file into my text editor.  

A small, but important point to notice here.  I wanted the first column ("Location") to be a comma-separated Lat/Long pair.  The easiest way to convert from a CSV file (where Lat and Long were in different columns) into a TSV file was to import everything into a spreadsheet, then write a small converter function that would concatenate the Lat + "," + Long into a single cell.  Then, save everything as a TAB-separated file (TSV).  That makes column one into a handy lat/long pair (like the first one:  42.74, -73.81 is the lat/long for Albany NY).  I also wrote an sum column at the end (that is, =SUM(A2:A26) which would add up all the snowfalls for the entire period), so the last column is total snowfall.  

Once I have this file, I can create a map of the total snowfall. 

I did the obvious search:  

     [ heat map tool ] 

and found the OpenHeatMap site.  It's really very easy to use--just upload your data, and you can make this map (showing the location of each of the snowfall summary rows, distributed on the map by the lat/long in column 1).  

Figure 1.  A choropleth map of NY state cumulative snowfall, first 2 weeks of November, 2014. Data from NOAA.

Strictly speaking, this is NOT a heat map (they admit as much on their website).  A heat map is a visualization of a matrix where each point in the array is color-coded.  

This figure above is actually a choropleth map.  Say that 5 times fast to get it firmly lodged in your brain.  Here's the difference:  A heat map is a colorized regular grid.  A choropleth map is a colorized symbol (like the circles above) or a symbolic element (such as a state in a colored state map) where the size or color indicate the variable value.  

In Figure 1 above, the circles vary in both size and color--that makes it a choropleth map. When you click on one of the circles, you can see the value of that location.  Example: 


BUT...  One of the options in OpenHeatMap is to blur/fuzz the circles.  This operation makes it LOOK like a heat map, but it's really just a fuzzy choropleth.  You have to be careful not to overinterpret what you're seeing.  The snow doesn't magically smoothly fall off the way this chart suggests.  Remember that ALL the data we have is total snowfall for a particular point.  This kind of chart suggests much more than is really here.  

Figure 2.  The same map as in Figure 1, just blurred a bit and using a rainbow from blue-to-red coloring scheme. 
Naturally, I wanted to see how to do the same thing in FusionTables.  So I created a new fusion table, and imported the data into there. After playing around with the values for a bit, I created this view. 

Figure 3.  A Fusion Table view of the same data. 

There you go, a slightly more sophisticated version of the snowfall map.  

If anyone is up for a Challenge over the Thanksgiving holiday, try this:  Can you make an animated version of the snowfall chart?  That is, could you make something like this?  

If so, Search On!  


  1. This was probably the most difficult challenge to me so far, and the most fascinating answer.

    Saved for future use.

    Minor correction: it's choro (from the Greek radical for "area"), not chloro. Not related to chlorophyll nor to the famous software Google Chorome.

    1. Thanks VERY much, Luís! I admit, it was a typo in my brain! (Fixed up the text per your suggestion.)

  2. An impressive set of results, Dan.

    I need lots of practice yet.

    jon tU

    1. We're here to give you the chance to practice. Go wild! (Seriously... to do this, one needs to practice now and then.)

  3. Unless different schools name map types differently (which, unfortunately, wouldn't surprise me much), it's possible that none of the maps shown on the post are choropleths. The first and second maps are "proportional symbol maps". The last map, looking like the "isoniph" or "isochionic line" map I was thinking of, is an "isopleth". At least that's what can be read in this very well done material about Map Types from the Institute for Geoinformatics / University of Münster.

    Here's a very lazy half-answer to the animated map challenge. It's just an animated gif composed of eight screen-grabbed images from CoCoRaHS Snowfall Map. It's too fast (I didn't tweak it right) and it doesn't show accumulated snow, just snowfall per day: http://makeagif.com/IvKzW1

    I've tried with a different non-interactive set of maps at the same website (CoCoRaHS), after manually tweaking the url address, and here's the slightly better-looking results (data from 12 to 22 November, since that was the period when it snowed heavily): http://makeagif.com/1FEVv3

    CoCoRaHs is a crowdsourced set of precipitation data, so the numbers might differ significantly from the official ones (personal measuring devices may not be standardly calibrated). Apart from that, it's very easy to retrieve their data, which is very neatly presented in a chart per day. Also, that type of colored dot map is, in my opinion, a great way to show what we want, and probably easier to do too.

    Last note: I'm not sure if "isoniph" is a good name, because it's way too similar to the already used "isoneph" (isonephs are for cloud cover).

    1. Here's the Fusion Map I've done out of COCoRaHS data of accumulated snow from November 9 to 19, ignoring zero values following the steps of Dr. Russell: goo.gl/AOKPjn. I felt the need to put my hands on it and learn by practicing too.

      As it can be seen if you compare with this post's map, data from CoCoRaHS is in fact similar to but not the same as the official data from NOAA.

  4. Dan,
    I got involved with moderating a large science community and that has sucked up my time so I don't get to participate in your weekly quiz as I would like. But I read them every week and this week was impressive.

    I use the skills I learned from your classes and this quiz every day to check references and to research new subjects. I could not do it without the proficiency I gained from taking the classes.
    Thanks for being such a good teacher and an amazing research guru.
    Happy Thanksgiving.

  5. Following the same logic of Google Flu Trends (mapping search terns that are "good indicators of flu activity"), we could try to find what search terms correlate well with snow.

    Instead of Google searches, another source of information could be Blogger or Facebook posts, or Twitter tweets. Simon Rogers has done a gorgeous animated interactive map of tweets using the word "snow" on February 13, 2014: t.co/gRc5Ak4RSZ. This map was done with CartoDB, which seems to be a very easy way to do good-looking maps. I signed up for their free service but didn't have time to try it yet.

    Rogers' whole weblog / website and his Twitter account are both filled with interesting data and amazing visualizations. His post about Twitter Mapping is probably worth a careful reading (but I'm postponing it for now).

    1. good find on Simon Rogers, Luís — his twitter has some timely related info for the snowy NE here in the US - the use of social/mobile media for mapping and data visualization is an interesting direction… although a bit at odds with the speed and fugitive nature of social media search/communiques…
      will make for interesting reading - twitter mapping

  6. I finally found a way to work with data in text file and moving them to CSV.
    This was my process:
    A. download open refine.
    B. Took data from Noaa. This link is in Dr. Russell answer.
    C. Opened text file in Open Refine.
    C1. Filtered results with only NY State.
    C2. Selected with a star only these rows. Here because I am new on this program, couldn't change all the "bad" data into 0.
    C3. Exported that project as a CSV, although not so sure, I need to verify because maybe it was TSV.
    D. Went to Google Spreadsheet, imported file. And it looks great, all the data at least in different columns.
    D1. With find and replace changed all the weird data. That is 00 999, and others.

    Know, I need to work with the data and create my heat map and later try the animated version. Also need to try Fred suggestion with add-ons in Google Spreadsheet.

  7. I made this one.

    I used the data from above.

    I was interested in both Michigan and New York lake effect snow so kept that data and deleted the rest. I used TextWrangler to move the data to Google Sheets. Eliminated the stations with less than 30 total accumulation because bigger is more interesting. :-)

    Transposed the data so it would be in the right format for the chart.

    So far I haven't found a way to overlay it to a map so the lat/long have not come into play. When I tried OpenHeatMap it gave me an error.

  8. I also did this one on Mapbox, but will admit it was a lot of guessing in CSS to get things to show up on the map. Not sure if it can produce a map over a chronological period or not.

  9. I'm still messing with this and have tried about 6 or 7 other options so far. I'll admit when they get into javascript and calling javascript libraries, I lack the background knowledge to put those together.

    I did a simple search for [ create map over time ] and have been working through the Search Engine Results pages. On page 5 of the SERP I found Tableau. Even though I have yet to create an animated map from it, it is by far the easiest tool I have come across. Yet in all of it easiness it could be the most powerful data visualization tool I have seen too. I dare say it is almost fun.

    And yes, it is FREE.

    So maybe some of you have the data all set. Give Tableau a try and see what you can make.

    1. Good find, Fred.

      I've tried Tableau with data from CoCoRaHS. The major problem is that it couldn't geolocate 401 out of 586 locations, contrary to Google, who found them all.

      Here's the map with the few points geolocated: bit.ly/somesnow.

  10. Here is what I finally got with Tableau Lake Effect Snow on Tableau. It didn't work for me in Chrome or Safari, but showed up in Firefox. Tableau only allows users to click through each day on the web version. On the desktop viewer it has a presentation mode and autoplay. I made this video using Quicktime of the auto play Tableau Screencast

  11. This is a follow up to my comment I made this weekend with links to the motion chart map I made with Tableau (at the time of writing this, previous comment hasn't been approved yet). I couldn't figure it out even after spending some time with their help file or tutorial videos, but something told me that everything was there to make it happen.

    I hit Google Search with [ tableau timeline map ] and went to Motion Map Chart with Tableau. Using what was written there and playing around with the variables I had is how I made the above Tableau Motion Chart.