SearchResearch Challenge (8/4/16): How many center-pivot irrigation systems do you see?

One of the most striking things about flying... 

... is looking down and seeing giant fields of circles on the Earth below.  If you've ever seen these, you might have wondered what they are.  They are NOT UFO landing sites, but are just fields that are being served by center-pivot irrigation systems.  Here's an example from Saudi Arabia (at 29.9653127, 38.3618227), where you can see that they are literally making the desert bloom (or at least turn a impressively dark shade of green.

Center pivot fields in Saudi Arabia at 29.9653127, 38.3618227 (P/C Google Earth)

When you're on the ground, there's a long, linear contraption that rotates about a central pivot hub, spraying water as it moves in a giant circle.  

which is driven from the center pivot that sends water down the pipeline.  

Center pivot hub, feeding irrigation water down the line.  

As you can see, these are pretty impressive systems.

Recently, as I was flying over the middle of America, I was wondering how the center-pivot systems in the US differ from other countries such as the Saudi system shown above in the first image. 

Here's an image from the middle of the US that has a lot of center-pivot systems: 

Center pivot fields in the US at 37.4359878,-101.6097197 (P/C Google Earth)

Obviously the soil is very different here. But what else is different here?  My curiosity leads to today's two SearchResearch Challenges:  

1.  (Easy) What's the most obvious difference between the US and the Saudi center-pivot irrigation systems?  (There are a lot of them.  Can you list one or two? How did you figure it out?)   

2.  (Hard)  How many center-pivot irrigation fields can you count in both of the images shown here?  Both images were taken from an altitude of 40 miles--but which location has more of the circular-disk-shaped fields?  (You should count but little and big fields.)  

The second question is a bit tricky.  Here are a couple of hints... 

A.  There are better ways than counting by hand.  Can you think what they might be? 

B.  I don't know of any obvious way to use Google to directly answer this Challenge.  I used Google in the process of solving it, but it was somewhat indirect. 

C.  Before you spend a lot of time trying random things, think carefully:  What would be a good way to approach a Challenge like this?  What would be the best approach?  (You should think to yourself:  What would Dan do??)  


As always, be sure to tell us HOW you solved this Challenge.  Tell us what steps you went through to solve this.  (For calibration, it took me about 10 minutes to find a method, and then about 2 hours to figure out how to use the method to come up with an answer.  

Search on!  

Answer: When you want just the headlines...

This was harder than most... 

... but everything is a bit harder in the City of Angels, as much hard-boiled detective fiction will bear out.  

But there are multiple solutions.  Today I'll just write about the simple Google News solution, and then later this week, I'll get around to showing you some other methods.  (It's slightly complicated because I'm traveling this week, which is why this Challenge answer is delayed.)  

Suppose I'm a reporter trying to understand how the Los Angeles City Council deals with gun-related issues.  Can you (expert SearchResearchers) tell me how to do the following? 

1. Can you search the major news outlets in the Los Angeles (LA) region for news articles over the past year that report on the City Council considering any kind of gun-related actions?  (Be generous here--if the council heard a report about the use of guns, that would count.)  

2. (Harder) Can you find the top 100 LA City Council headlines on guns, and then extract the publication dates to create a week-by-week histogram of when these articles were published?  (This is a two-step challenge: (a) find and extract the dates, (b) put the dates into a spreadsheet and create a histogram showing the number of publications on this topic by week.)  

As I mentioned last week, the problem here is to search JUST on the headlines.  How can you do that on Google? 

First of all, remember that there IS a Google News service.  So what happens if we search for: 

     [ Los Angeles City Council gun ] 

 You'll get a fairly unfiltered set of news articles with these words... which aren't very specific.  I'd say this isn't a great way to go--it's too open-ended, and the search terms aren't always in the headlines.  This is because default News search searches both the headlines and the body text.  

The big trick about searching Google News is to realize that there's a hidden dropdown widget in the News search box.  See below:  

It's only when you roll-over that downward pointing triangle that you realize what it's for: 

Ah ha!  It's the apparently-missing-and-well-hidden Advanced News Search UI! 

If you click on that, you'll get lots of options that would be handy for a Challenge like this: 

I've filled out the form to search for city and council  and gun within the headlines of the article between the specified dates.  

Note that I also added a Location at the bottom--here I put in Los Angeles.  

And the results come back: 






You might wonder why the 4th article is considered "in" the Los Angeles area.  It's because that's where the website, OpposingViews.com is headquarted.   The Los Angeles Times and KABC-TV are obviously correct, but the "Location" filter works off of the location of the organization that's writing, rather than the location that's being written about.  

(Note that this is one of those things what will get better over time as entity identification improves.  For for the moment, we have to live with this as it is.) 

Now, if you do this same kind of "headline search" with a different query that returns fewer results, it means you might consider what OTHER papers not-in-the-area might well write about, but be headquartered somewhere else. 

So let's run a query for [ county courthouse ].  If you search only in Los Angeles, there are only 5 results.  But change the location to "California," and you'll get a superset of all the headlines written about the county courthouse from news organizations within California.  

Now we see a LOT more results (around 60): 

As you'd expect, there are more slightly off near-hits here.  The "Knox County Courthouse..." article is from SFGate (in San Francisco), but it's not about California.  We'd have to filter through these results as well, in order to get to the kinds of news stories that we really want.  
But at least you've learned a new way to search JUST the headlines of News stories.  

However, I have to zoom to the airport just now... I'll answer the question about how to extract the dates and create the histogram tomorrow, along with the Search Lesson. 

Stay tuned, and search on! 

Search Challenge (2/17/16): Where are people moving to and where are they moving from?

Where is everybody moving to? 

It's a big political year in the US, with the presidential race heating up, preparing for the election this coming November.  

And, as usual, the candidates are debating this and that, mostly without much data to support their various assertions. You might hear a number or two, but are these numbers real?  Or do they just make them up?  

Our SearchResearch blog is all about data and getting to the bottom of things--we're here to help find data and answers to complex questions!  

One of the big debating points is whether or not there's a mass immigration into the US across the border with Mexico.  (People never seem to worry about the much bigger, much less defended border with Canada... but I digress.)  

And, more broadly, there is a human tide of people fleeing Syria and many part of the Middle East.  

The big Challenge for this week is to get some kind of data handle on all of these mass movements of people.  Where do they start?  Where are they going?  And perhaps most importantly, how do you know?  

Our Challenge this week is straightforward to express, but not so straightforward to solve: 

1.  During the past year, what have been the large movements of people across the world?  (We're interested here in people who are moving from one place to another for economic, political, or safety reasons--not people going home temporarily for the New Year's or Thanksgiving Day celebrations.)  

In particular, can you: 

A. Find data sources that tell us how many people moved from location X to location Y in 2015? 

B. Is there a good way to visualize that data?  (Giant tables of numbers aren't all that descriptive, there must be a better way. Can you find a good one?)  

C. How do you know that these numbers are right?  And to what extent can you believe them?  Why?  

Obviously, there's a whole world full of people moving.  If you want to focus on just one area, the most important for the US is the movement of people from Mexico into the US (and from the US into Mexico!).  If you want to understand the Middle East, look into movements of people from Syria (where are they going?), and Turkey (where are they coming from?).  Or, if you're from India, what outflows and inflows of people are moving from India?  

This time, we're really interested in HOW you found your data sources, and a bit about WHY you believe this particular source(s).  During the political silly season, you'll hear all kinds of crazy numbers--which ones are reasonable estimates, and which are just beyond belief?? 

Search on!  

Wednesday Search Challenge (August 31, 2011): Easiest way to animate the visualization

There are no end of hard problems to ask.  Most of the really good ones aren't quite suitable for this blog--what is the nature of life?  why is there something rather than nothing?  


So I tend to pick problems that have some kind of observation about the world.  Someone will ask me a question ("Is synesthesia consistent across people?") or I'll notice something odd about the world ("What is that anode thing in the middle of the street?") and that will lead to a search challenge.  


For today's question I want to pose a more mundane, but in some ways even harder, much more pragmatic question.  (And I ask this in the hope that you'll come up with a solution that I haven't thought of!)  


Remember last week's question about the words kayak and tint?   I thought so.  


When I saw the data curves, I immediately wanted to make an animation of the US vs. New Zealand data showing the maps of both US and NZ growing darker/lighter with the value of each query volume.  


This is the kind of practical question someone might ask a data analyst. Can you make me a cool animation that does... this cool thing?  


So the challenge is this:  Figure out how to make an animation of US vs. NZ that pulses the color of the country from white-to-black with the values of each query.  I'll give you the data so you don't have to hassle it yourself.  


Link to Google Spreadsheet with data. 
Link to XSL version of data. 
Link to CSV version of data. 


Each version is 104 rows of data, one for each week of 2009 and 2010.  The values vary from 0  - 100 , with 0 being the lowest query volume.  


Go ahead, so us your animation!  (And let us know how you did it!) 




Search on! 

Answer: How many planes from SFO / day?

Last week I asked a question that should be pretty simple to answer:  How many regularly scheduled flights depart SFO (San Francisco airport) on a typical day?  


The hard part here is to find a reasonable data source of flights from SFO.  Here's what I did: 



First search for [ faa flight departures ] 


When I scanned the results, I quickly find that the FAA has an "ASDI Data Feed," which shows all of the flights that the FAA tracks, and that wouldn't be a bad thing to use, but I only want to do this once from one airpot, not many times over and over, so that seems like overkill.  


In my reading of the results I also learned about codeshares.  Ah, that's right, any listing of airline flights will have the same flight listed as being from multiple carriers.  Since I wanted the number of planes departing, I need to exclude the codeshare flights.  Don’t want to double or triple count these, so I want to get a singlecode for 1 plane.  


After then searching for [ SFO flights listing ] I find that FlightStats.com allows you to search for all departures by the hour!  Although it's inelegant (and I wouldn't want to do it this way more than once), I can manually copy/paste all of the departures from SFO into spreadsheet!  Luckily, FlightStats also has a "Remove codeshares" button which lets me skip the post-data download step of removing dups.  


It seems painful, but the truth is that I was able to grab the 18 screen-fulls of data in about 3 minutes and paste it into my spreadsheet.  Great!  

After spot checking to make sure that I really DID have all the flights, and that there really were no codeshares, and that I hadn't accidentally duplicated data somehow, I was ready to go.  Finding the number of flights was then just looking at the last row in the sheet:  there are 583 regularly-scheduled flights / day out of SFO (at least for July 12, 2011).  


But I couldn't stop there!  From here, it was easy to write out a CSV file and then import that into Google Fusion Tables in order to use the map visualization tool that I know is there.  A quick import, and then a click on "Visualize" and then "Map" yields: 

Interesting, is it not, that there are more flights to more cities in China that to Russia, Africa and Australia combined?  

And for my last quick look at the data, I computed a histogram to see the number of flights departing by hour of the day.  As you can see, it's quiet after midnight, but pretty steady for most of the day.  

And finally, this chart that buckets departures by 30 minute buckets... 

So now you know the quiet times for departures at SFO.  I've flown enough times out of SFO to say with some certainty that this is probably right.  

People who left comments (Hans, Julia and gasstationswithoutpumps) all had reasonable guesses, but they were estimates from other people's summaries.  That's a valid method when you can't get to the actual data, but when possible (as in this case), I like to get the raw numbers and do the analysis myself.  (Still--Hans got extremely close to my number! Nice job.)  

There are a couple of search lessons here.... 

1.  When you can, get the raw data and do your own analysis.  Other people's analyses often contain assumptions that you can't verify (and they might not even explain).  Be careful! 

2.  A really important part of the search analysis process is learning stuff along the way.  This is such an important concept that we'll have to talk about it in more detail in a future post, but I wanted to highlight it here.  I was reminded about the codeshare problem as I read through the initial search results... and that let me know how to frame my subsequent searches to be more to the point.  

Keep searching! 

Postscript:  I also should mention here Aaron Koblin's wonderful visualizations of air traffic patterns. It's not quiet what I was asking for, but you might be interested in seeing them.   

Here I zoomed into his flow patterns around SFO.  The labels are mine to add a bit of clarity.