Search results sorted by date?

I wanted to highlight an interesting discussion that's been happening in the comments of last week's Challenge.  

Regular Reader Diane asked a great question: 

"... [is] there is a way to order Google search results by date?I did a search on ["search lessons" site:searchresearch1.blogspot.com ] to get a list of the search lessons you provide in the various challenges. I'd like to be able to order my search results by date, with the most recent one first. Is there a way to do that?"

After she asked that question, Regular Readers Ramón and Remmij pointed out that Diane has two options. 

First, if it's just the blog posts that Diane wants to see in chronological order, the simplest way is to look at the "Blog Archive" on the right hand side.  You can scroll down now to see them.  It looks like this:

As you can see, the entire history of SearchResearch is there, all the way back to 2010.  

(Has it really been 7 years of blogging?? My how the time flies.)  

You can also search for specific topic within all of the blog posts by clicking on the search box in the upper left: 

.. which you can then sort by date.  

HOWEVER... oddly enough, this search tool only shows you the top 6 hits.  If you do a site: search like this, you'll find many more!  

Here, I'm searching for every blog post where I use the word "polymath."  There are 22 results over the past 7 years.  

But perhaps more to the point, you can date restrict any search results by using the time filter tool.  

To do that, click on the "Tools" item (circled in the above image), then select which option you want to restrict by.  

This isn't quite showing the results sorted by time, but you can get pretty close.  Here's my search limited to just 2015.  First you specify the date range for acceptable results by entering just the dates for 2015.  

Then, when you click on "Go" you'll find that you get just the results from the blog that were published in 2015.  

This isn't quite what Diane asked for, but it's pretty useful! 

Search on! 

SearchResearch Challenge (6/14/17): What's difficult for YOU to find?

By now you know... 

... what kinds of research questions that come up for me.  In the past couple of months we've done research on: 

     - finding shadows in ball parks 
     - what kind of mirages cause ships to apparently float in mid-air 
     - how to find tweets from a particular place 
     - finding cartoons from a not-quite-correct description 
     - how to build an interactive widget for the island-viewing problem
     - exploding seeds 

So you know that I'm interested in the kinds of questions prompted by photos like this:  

Major General George A. Custer, officer of the Federal Army. (circa April 15, 1865)
Brady National Photographic Art Gallery (Washington, D.C.).

Did George Custer have freckles?  

(Or is it an imperfect in the photographic process?  Or is it some other skin condition?)  

Or... 

Where did that crazy uniform design come from?  Was that standard Federal Army issue?  

But that's not the question for today.  Today's Challenge is about what kinds of SearchResearch questions come up for YOU in your average week.  Here's your Challenge for this week: 

1.  What kinds of things do you find tough to research?  In an average week (however you define that),  what topics and questions do you find yourself trying to research?  

I'm asking this question broadly (in my Facebook posting, on Twitter, G+, through surveys, and other venues).  

Next week I'll pull together the answers that have come my way, and I'll try to identify themes and topics.  (And, in the process, try to come up with some future SRS Challenges that will be based on what I hear from you.)  

Search on!  (And tell me what kinds of things you do research for, and what makes it difficult!)  

Answer: Finding shadows in ball parks?

There's simple.. and then there's simple...  

The FIRST version of this week's SRS Challenge was a bit too simple.  Finding shade at Yankee Stadium (along with several other stadiums) is pretty easy:  lots of fans have written up guides to sun and shade.  That's a handy thing to remember.  

So a query like: 

     [ sun shade yankee stadium ] 

will find notes from all of the fans, documenting where to sit to be in the shade.  
Too easy.  

So let me rewrote the Challenge just a bit to make it more general... 

1.  My brother has changed his mind, and now he's going to see a baseball game at the Red Wings baseball stadium in Rochester, New York on June 11th.  The game starts at 1PM.  Where can I get a seat that will be both near the field and in the shade for the whole game?

Here's  Frontier Field, located in downtown Rochester. Founded in 1899, the Red Wings are the oldest continuously operating sports franchise in North America below the major league level. Located at 43.1551745,-77.6196905 (lat, long). 

More to the point, I would like a method that lets me figure out the shadows for anyplace--be it Frontier Field in Rochester, or a lesser-known place like Kezar Stadium in San Francisco that doesn't have a stable fan base to write out such notes!  

As we've discussed before, the ideal solution would be to find a tool that let's you figure out the sun and shade for any place.  (And an even BETTER solution would let you create a 3D model of the stadium)  

So my goal is to find such a tool.  My first query was: 

     [ finding shadows tool ] 

which led me to FindMyShadow.com -- a web tool designed exactly for such a thing!  

So let's go back to Yankee Stadium and figure out where the shadows will be for the game.  (You can do this same process for Frontier Field...) 

When you go to the site, you can selection the location (see the lat / long box in the upper left), select the date (calendar).  The sun position chart shows sunrise (the green arrow on the right, showing it rising in the east) and sunset (the red arrow on the left).  

Note that you can also set the Timezone (which for this time of year is EDT -- or UTC-5 (which is the same as "5 hours before GMT").  

Once you've got these location and time set up, you can bring in an aerial image (of Yankee Stadium), and THEN you can draw in the relevant walls of the stadium. 

You click on "Background Image: (choose file)" to upload the image of the stadium. 

Then you can move the brown boxes to approximate the edges of stadium.  (Drag the corners to move them side-to-side, and click on the green center dot to rotate the rectangle.) 

Note that I filled in the height parameter (bottom line of the interface--it's 123 feet high--I looked it up), so the gray shadows show the actual shadows at that time (1PM) in this stadium (the model of Yankee stadium). 

With this tool, you can model shadows for anyplace you might like, such as Estadio Azteca (located at 19.303204,-99.1516849), Kezar Stadium, or Frontier Field in Rochester!  

Estadio Azteca, with a difficult shadow line.  Be sure you sit in the shade! 

Of course, if you want, you can grab a seating chart image, then rotate, scale it, and make it transparent to see which seats you might like to have.  Here's a closeup of me doing that.  I find where the shadow is, then pull up the translucency so I can find out what section and seats are there. 

When you play around with this (as I did), you can see that most of the seats along the first base side of the field (lower left) are in the shade starting at 1PM.  But advancing the clock to 3PM, you'll find that some of the seats near home plate start to get some sun!  

In any event, now you know how to figure out the sun orientation (and shade patterns) for basically any place in the world.  You can now go to that sports event confident that you'll be in the shade the entire time.  (Or not, as you wish.)  

In the process of playing around with FindMyShadow.com, I also ran across a really interesting article on the physics and factors involved--"Lost in the park: The physics of ballpark orientation," which is a fascinating read.  

Search Lessons 

There really is only one giant take-away from this week's Challenge, but it's an important one: 


1.  ALWAYS look for a tool when you're trying to do tool-like searches.  A good query is to search for the basics (e.g., [ sun shade .. ] and add in tool  Yes, there are fan tables of sunshine and shade, but many of them are for specific days in the year and for a specific location.  FindMyShadow.com is purpose-built for this kind of thing.  It's almost certainly better than any table you're going to find.  (Although it is a bit more complicated to use!)  

That was fun.  Up next--even more fun in searching out the unusual and obscure.  


Search on!  

Finding shadows in ball parks? An update. Version 2.

When I said it was going to be simple,
I didn't mean THAT simple!  

As Regular Readers quickly found out, Yankee Stadium (along with several other stadiums) offers online guides to sun and shade.  That's a handy thing to remember.  

But I want to up the ante just a little bit, and propose that we try to find a more general solution than that.  

So let me rewrite the Challenge just a bit: 

1.  My brother has changed his mind, and now he's going to see a baseball game at the Red Wings baseball stadium in Rochester, New York on June 11th.  The game starts at 1PM.  Where can I get a seat that will be both near the field and in the shade for the whole game?

Here's the picture of the stadium where the  Rochester Red Wings play.  They're a minor league baseball team based in Rochester, New York. They play in the International League farm team for the Minnesota Twins. They  play in Frontier Field, located in downtown Rochester. Founded in 1899, the Red Wings are the oldest continuously operating sports franchise in North America below the major league level.

To help you out, the field looks like this (see below), and is located at 43.1551745,-77.6196905 (lat, long). 

(I did a quick search and didn't find an online guide to sun / shade and seats at this particular place.)  

So can you help me figure this out?  The ideal solution could take any stadium and predict where the sun and shadows will be at 1PM on June 11.  

Search on! 

SearchResearch Challenge (6/7/17): Finding shadows in ball parks?

Take me out to the ball game... 

Outdoor sports stadiums sometimes have a little trouble with the sun.  

Yes, it's great to be outdoors watching the players doing their thing, but it's also sometimes a hassle, especially if you're watching a long baseball game in the hot sun, or a player trying to track a ball on a long throw as it passes from full-sun into full-shadow.

Stadiums outdoors have issues with sunlight and shadows.

My brother asked me a version of this Challenge earlier this week.  Much to my surprise, it was easier than I thought it would be.  Here's my version of what he asked me: 

1.  I'm going to see a baseball game at the New York Yankees baseball stadium on June 11th.  The game starts at 1PM.  Where can I get a seat that will be both near the field and in the shade for the whole game?

As I said, I was surprised at how simple this was to solve.  Can you find the answer that I did? 

The Search Game is on!  (But put on sun screen.)  

Answer: Nautical mysteries?

Odd stories:  How can you verify / follow-up?    

That's really the question this week--when someone tells you something odd, how can you (quickly) follow-up and check out the veracity ("truthiness") of their claim?  

Here are the slightly whacky claims I heard this past week.  In each case, what's going on?  True.... or not? 

1.  I keep reading about people going whale watching in Lake Superior.  Can I really see whales in Superior??  (It is, after all, a great lake.  Could whales possibly live there?) 

Let's start with the obvious query: 

     [ whales in Lake Superior ] 

You'll get something that looks like this.  (I added the red numbers, obviously.) 

Let's look at each of these results. 

#1 looks like the real thing.  Not only does it look like there are whales in Lake Superior, but there's even a whalewatching tour boat industry.  You can even buy a t-shirt!  But let's click on that link.  Here's what that landing page looks like when you click on the link (note that I trimmed the logo off the top in order to have space to fit this all in).  

This ALSO looks pretty good.  There's a place for reported whale sightings, and a list of past sightings.  AND there's even what looks to be a whale's tail flukes coming up out of the water.  

What wait a second.  If you click on the "confused" link (on the right), it takes you to a page (which happens to be #2 above) where they admit that this is all a giant hoax for fun.  Whales don't live in fresh water... 

Despite finding whale fossils near the shores of Lake Michigan (#4 above), those fossilized bones are from quite a while ago when things were... different.  

And of course when you look at the Snopes.com article (which happens to be about Lake Michigan, but it's the same story), they also debunk the claim.  It was a rumor started for fun.

It doesn't matter if there's a YouTube video (#5).  A quick search for: 

     [ whales in fresh water ] 

leads to a number of articles that point out that while some dolphins can live in fresh water (such as the Amazon river dolphin, Ganges river dolphin and Indus river dolphin), whales can't tolerate fresh water at all.  

So no, don't count on whale watching this summer while touring the Great Lakes!  

2.  I was reading a bit o' Irish history and happened to read about a place called Clonmacnoise, where, once upon a time, a few of the monks apparently saw a sailing ship fly past in the air.  How would such a thing be possible?  Can you figure out any plausible way that people could see ships in mid-air?   

This is slightly more complicated--how are we going to start? 

Let's break this up into two parts.  First, can we find the Irish history about ships sailing through mid-air?  And second, what could cause someone to see a ship (any ship) in the air? 

For the Irish historical context, I started with: 

     [ Irish history ship in mid-air ]  

Remarkably enough, the first hit is to an article on this very topic!  The page is about the book The Wonders of Ireland by P. W. Joyce, 1911.  In that book is told the tale of a ship that passed by a village:  

"On a certain day when Congalach, king of Ireland, was at the fair at Tailltenn with a great assembly of the men of Erin around him, he looked upwards and saw a ship floating about in the air high over his head. While the king and his people were gazing at this strange sight and following it with their eyes in silent wonder, they saw one of the crew come forth and cast a dart at a salmon, which appeared also suspended in the air near the ship. He missed his mark, and the dart fell to the ground in the presence of all: so the man walked out over the side of the ship, and floating gently down towards the spot where his spear was lying, he stretched forth his hand to pick it up."

Okay.  So it's clear that amazing things happen in Ireland, or at least amazing stories are told...   (And for a really interesting and learned article on this topic, please read Voyagers in the Vault of Heaven from Material Culture Review. Who knew?)  

But for the second part of our question, I did a search for: 

     [ appearance of ship in mid-air ] 

The first hit is to the Sun newspaper's article with a photo of a cruise-ship apparently hanging in mid-air!   (Link to Imgur image.) 

Unfortunately, the Sun (UK) newspaper isn't exactly the most reliable source on Earth. 

(How did I know that?  As always, I checked the source by doing a query for the name of the newspaper.  That way, I learned a lot about the source of this interesting image.  It seems to be a real image from an unreliable source.)  

BUT.. the article does have some value to us as researchers.  

The headline of the article is "ITS A FLOATER! Mind-bending photo of a cruise ship hanging in MID-AIR could explain how the Flying Dutchman got its name."  

And the subhead is:  "The illusion is known as a Fata Morgana, where light bent across the atmosphere creates projections of images."

Now that's interesting.  What is a Fata Morgana?  

     [ fata morgana ] 

This query shows us that it's a kind of optical illusion that is seen in a narrow band just above the horizon. Named for the sorceress of Arthurian legend, Morgan le Fay, after a belief that these mirages were fairy castles in the air or false land created by her to lure sailors to their deaths. 

Fata Morganas significantly distort the objects on which they are based, so much so that the object is often completely unrecognizable. 

In truth, the Fata Morgana is an optical phenomenon that happens when  rays of light are bent as they pass through air layers of different temperatures, often along a steep thermal inversion where the light is bent.  

Here's a series of images of our favorite islands, the Farallons, taken from a large distance away (each is taken a few seconds apart near sunset).  

P/C Wikipedia and photographer Brocken Inaglory. CC BY-SA 3.0  

But can a Fata Morgana create the illusion of a flying ship?  By searching for: 

     [ fata morgana flying ship ] 

I found this article in Wired (linked to by both National Geographic and Scientific American) that claims is very possible.  

That makes sense, as the last image in the Wikipedia article illustrates.  This image is from the Frank Leslie’s Illustrated Newspaper (May 13, 1871); Department of Rare Books & Special Collections, University of Rochester Library and shows a huge Fata Morgana as sketched from a hill in Mount Hope Cemetery, Rochester, New York. 

from: Wikipedia Media Commons.  

And the reason THAT struck my eye is that I used to live right next door to this cemetery.  I used to run in the cemetery almost daily, and trust me, you can't see Canada from there.... unless there's a Fata Morgana in play.  

Here's a Google Maps image showing the cemetery (and incidentally, where I lived, in one of those brick apartment towers just to the south of the cemetery).  You can see Lake Ontario to the north.  Toronto is 45 degrees to the left (NW) from the cemetery.  And I have to admit that I would have been very surprised to see the Canadian shoreline rising up from the horizon like this... 

If a fata morgana can make a coastline seem to float in the air, a boat would be easy.  

In any case, I think we've figured it out.  A Fata Morgana can make a ship near the horizon seem as though it's actually flying in mid-air.  

3.  Since I'm in landlocked Switzerland, someone happen to mention that Switzerland has a navy.  I've only ever heard that term used perjoratively, so I wonder: Really?  Does Switzerland really have a navy?  If so, where is it?  

To start, I asked the simple query: 

     [ Switzerland navy ] 

and found a couple of Wikipedia entries describing a small flotilla of military patrol boats that's actually a branch of the Army.  They not only do patrols, but also search-and-rescue.

These days, the Swiss Navy branch of the Swiss Army has Aquarius-class (Patrouillenboot 80) riverine patrol boats, on lakes Geneva, Lucerne, Lugano, Maggiore and Constance. 

Such collections of vessels in landlocked countries are sometimes called "brown-water navies."  

GlobalSecurity.org (a major web site that describes military equipment and inventory by country) describe the Swiss Navy as "Even if landlocked... [Switzerland] ...has a small navy of sorts." 

Switzerland also has a long and rich tradition of having a merchant marine flotilla.  While these aren't full naval fighting vessels, the merchant marine was set up to keep Switzerland in provisions during the Second World War.  This has continued ever since.  

So yes, they have a navy--a smallish, brown-water navy that patrols the rivers and lakes of Switzerland (especially those lake that have international boundaries in them).  

And the phrase "Admiral of the Swiss Navy" is a pejorative term used to describe someone who has a very inflated sense of self-worth.  Interestingly, this term is used only only in the US forces since the Swiss lakes flotilla has no admiral, as it is only a company-sized unit of the Swiss Land Forces.

Search Lessons 

When you read something that seems a bit extraordinary, it's worth spending just a couple of minutes to track it down and verify the wild notion.  Today we learned that: 

1. One query is often enough to find enough information to verify (or disqualify) a sketchy piece of information.  For instance, whales in the Great Lakes sounds reasonable, until you do that one query, and by looking at multiple results, you can quickly find that this is a bogus idea that was started as a joke... and got a little out of hand.  

2.  Even unreliable sources can have clues that are useful.  In the case of the "flying ship," I don't trust the Sun to have much reliable information.  But in this case, they gave us an important term--fata morgana--that let us find good information from other sources.  Verify your sources, but don't let them blind you to every possible clue! 



Thanks to all who contributed comments this week.  Nice job, everyone!  

Stay tuned for Wednesday's search Challenge:  I've already worked it out (which is unusual for me), but it's a very fun one to do.  It's also really useful for planning your summer.  

See you in a couple of days. 

Search on! 

SearchResearch Challenge (9/31/17): Nautical mysteries?

Every so often you hear a few stories that just seem to go together.  

This week, even though I'm teaching in Switzerland (maybe because I'm in Switzerland), I hear three stories that seem a bit too odd to be true.  Can you help me figure out if these are real stories or not?  

Here they are.  In each case, what's going on?  True.... or not? 

1.  I keep reading about people going whale watching in Lake Superior.  Can I really see whales in Superior??  (It is, after all, a great lake.  Could whales possibly live there?) 

2.  I was reading a bit o' Irish history and happened to read about a place called Clonmacnoise, where, once upon a time, a few of the monks apparently saw a sailing ship fly past in the air.  How would such a thing be possible?  Can you figure out any plausible way that people could see ships in mid-air?   

3.  Since I'm in landlocked Switzerland, someone happen to mention that Switzerland has a navy.  I've only ever heard that term used perjoratively, so I wonder: Really?  Does Switzerland really have a navy?  If so, where is it?  

I don't think these are really hard, but they're questions that all came up for me this week.  What's your advice?  Should I trust these statements? Or are they... fake stories?   

Search on! 

Answer: Finding tweets from a particular place

Geolocating Tweets... 

People tweet about darn near everything.  That's handy for you, if you're a SearchResearcher.   

I figured out that my friend's house was NOT on fire with geolocated tweets.

 

This week's Challenges:  

1.  When the Google I/O event happened last week, I wanted to see what kinds of things were being tweeted about.  Unfortunately, not everyone adds the #GoogleIO hash tag to their tweet, and sometimes people add the hashtag when they weren't really there.  Can you find tweets that were posted from INSIDE of Google I/O 2017?  

We want to search for tweets that are geo-located to a particular place.  In this case, the location of Google I/O 2017.  

And where was that? 

     [ Google I/O 2017 ] 

quickly tells you that it was held at the Shoreline Amphitheatre, right by the Googleplex.  (lat/long: 37.4263042,-122.080828)   

You can search for Twitter's Advanced Search UI (by doing this search): 

      [ Twitter advanced search ] 

but you'll find that you can search for towns, but not specific geolocations.  

I next did a search for: 

     [ Twitter geocode search ] 

and quickly learned from a number of sources (e.g., ResearchGate and ThoughtFaucet) that you could use the regular Twitter search box and drop a geocode in there following the pattern:  geocode:LAT,LONG,RADIUS -- where you enter the latitude, longitude, and radius with a units (e.g., 10km) appended.  ... and no spaces!  That is, the modified query is now: 

But note also that Twitter returns tweets within your given radius by using their own internal methods to geolocate tweets. They use a combination of device/gps coordinates (but ONLY if the user has opted into providing their location) and the user provided profile location.  Usually, "user-provided profile location" will be a city, and not often an amphitheatre location.  

2.  Obviously, you'd like to be able to restrict your tweet lookup by time and date.  Can you find those INSIDE tweets from Google I/O that were posted only during the days of the event?  (May 17 - 19, 2017)  

Let's go back to Twitter's advanced search page.   Here they have the ability to search for Tweets by date range.  When we fill out that date range, we see that it creates a query to Twitter with this in the middle of the URL: 

      %20since%3A2017-05-17%20until%3A2017-05-19

That's the date-range specification for a Twitter query.  (For the curious: %3A is the URL encoding for the character ":" (colon) while %20 means " " (space).)  

This means the whole query really is: 

         geocode:37.4220041,-122.0862515,5km since:2017-05-17 until:2017-05-19

This is generally a good trick to know:  Many search engines pass through arguments through the URL.  If you decode the URL into its parts, you can often replace it with your own value and get exactly the results you want; in this case, we limit the results of the lat/long geocode: query to ALSO include the date limits.

If you edit the URL for the geocoded query (see above), and add to the end of the URL the date-range limiter shown above, you'll get a result like this (I put the dashed oval around the date-range restricted part of it).  

Obviously, if you were NOT doing a geocode query, you could just do this with the advanced search UI.  

OR if limiting the search results by city was okay, then you could do both the city and the date restriction by the advanced UI.  That's actually not a bad way to do it.  Yes, you get MORE than just Tweets from Shoreline Amphitheatre, but they're all pretty good. 

3.  Since you can find the location of a tweet, is it possible to make a map of tweets that are posted from the city of San Francisco during a single day? How would you do this?  (I'll post the best maps in the blog next week.)  

There are many ways to do this, but let's start with the simplest:  Find an already existing solution. 

My first query was: 

     [ tweet map ] 

This led me to Mapd's Tweetmap demo.  Here's an image of the map for tweets with the hashtag #GoogleIO, zoomed into San Francisco.  

Here you can see (on the very bottom) the tweet volume by day.  The big peak on the right hand side is May 17, 18, and 19.  


But I was hoping for a way to search for tweets (where I could guide the tweet selection), and then put them onto a map that I could then edit as I wanted.  

To do that, I knew I'd need a tool that could collect the tweets (given my query), and extract the geocode, and then put them onto a map in some way.  

In truth, I poked around for quite a while, trying out different options, finally settling on this.  

I figured that someone probably built a Google Spreadsheets add-on that would grab the tweets.  So my search was: 

     [ Google spreadsheets tweet collection ] 

There are several.  After testing a few, I found that Twitter Archiver seems to fill the bill.  It's an add-on to Google Spreadsheets that lets you write a Twitter query (including using geocode:) and then it runs every hour and updates the sheet.  But as I found out (after a couple hours of debugging), it won't let you search for tweets from a specific place OR a specific location.  

But it will search for recent tweets and drop them into your spreadsheet... from which you can then do your own processing.  Here's what I did with it.   

You add the Twitter Archiver to your spreadsheet, and then create a "Search Rule" to specify what you want to download into the spreadsheet.  

The add-on then runs this query every hour and adds new tweets into the spreadsheet.  It then looks like this (notice that the dates on the far left are from today, NOT from the dates of the conference--if I'd run this last Wednesday, I'd have all of those tweets!) I've blurred out people's names and IDs. 

Don't be mistaken: this is NOT a regionally restricted search (no matter what it says).  But the results are coming from all over.   

This is pretty good, but we now need one more step to put these out onto a map.  Luckily, I remember that Google's My Maps lets me import a CSV, including lat/longs and content.  

I created a map and then imported the spreadsheet into my map, selecting the "place" column for the pin to drop.  

Interestingly, here's that map: 

This is close, but not quite what we had in mind.  FWIW, I tried out a few other scraping tools and found that none of them (at least of the ones I tried) quite have the ability to search by specific geocode.  Clearly Twitter has the ability--but the results aren't making it into the scrapers.  

This probably means that I'll need to go write some real code in order to make this work.  I'll try this later this week and report back to you. 

And we now know how to make a map with pins for all of the tweets... once we get them.  

But at least we found a commercial tool that nearly there!    

4.  For completeness, it's useful to know how many tweets come with a geocode. Can you estimate the fraction of tweets that are geocoded?  (I ask because you need to know what fraction of tweets you're NOT seeing when you do a search for tweets posted from a particular location.  If you're only picking up 5%, then that's a very different story than if you're seeing 95% of all tweets.)  

If you search for: 

     [ fraction geocoded tweets ] 

lead me to a Quora article by Ryan Gomba claiming that around 1% of tweets are geocoded. Other sources give similar number:  "less than 2% of tweets are geotagged" (Laylavi et alii, 2016), and it's clear that the number is growing over time, but is still quite small.  In an Arxiv.org paper, from 2014, 3 researchers from IBM give the percentage as 0.7%, but also note that they have developed an algorithm that can tweeters' home location fairly accurately by looking for other contextual information (e.g., explicit references to locations over time, noting time of day, and correlating posts with other, similar posts, in other social media).  

So it's not a large fraction of all the posts.  Still potentially useful, but it's not going to be anything like a statistically useful sample for most purposes.  

Search Lessons 

Well this was a tough one--I spent WAY too many hours on this trying to get various Twitter scraping tools to work.  

1.  Remember that many web services have their own advanced search capabilities.  This is a really useful thing to know: you can often do a lot by using a company's search tool, rather than counting on Google to figure out all of their internals (like geocoding or time/date filtering).  Search for it! 

2.  If you real the URLs carefully, you can often modify them to suit your own needs.  As we see above, with a little sleuthing (and URL surgery), you can get their search tool to do what you want (such as combining a geocode: search with a until: and before: search time restriction. 

3. Sometimes tools don't do quite what you want--test them!  The big moral of my multi-hour scraper testing is that you have to test your tools very, very carefully to make sure that you're getting what you THINK you should be getting.  (I spent a couple of hours trying to figure out why the maps weren't correct... only later did I realize that although I gave the scraper my geocode query, it wasn't actually returning those results.  Moral:  Debug your system with data you can easily validate!)  

Okay... Onward to the next Challenge later this week.  (It will be much easier, I assure you.)  

Thanks to everyone who wrote it.  This was tricky!  

Search on!