Friday, December 30, 2011

How long does it take to find an answer?

This week's challenge ("Who IS that woman?") wasn't especially tough. It was interesting, but not particularly hard to do.  But there are some people who will disagree with me, and therein lies a great tale.  


Roughly 12,000 people read the challenge, and if you look at the 150+ comments I got on this, it seemed to capture many readers interest. People who solved the problem reported time-to-solution ranging from 5 seconds to 60 minutes.  I went through all the comments (and personal emails) to collect 94 reported times.  Here's the chart of how long people said they took to find the answer.  


The times on the bottom are in minutes.  You can see that 22 people took less than 1 minute (in many cases, less than 30 seconds) to discover that our "fascinating woman" was Lou Henry Hoover, wife of Herbert.  Or, to put it another way, all of the people in the green part of the histogram (who took 4 mins or less) were almost exactly half of the total.  In other words, 50% of everyone who solved it found the answer in less than 5 minutes.   (And yes, I know this is a highly biased sample.  It's only people who solved it AND commented on their solution.  For all I know, there are thousands of people who couldn't solve it at all, or took more than 60 minutes and were embarrassed to report their times.)  


The other way to look at this is that 50% of those-who-solved-it took 5 minutes or MORE (that's the red part of the histogram).  


This kind of pair of curves is fairly common in my research.  There's a group that gets to the problem solution fairly quickly, and another group who take longer... 50% took less than 5 minutes, while 50% took more than 5 minutes.  Another interesting fact is that 14 people in this sample took more than 10 minutes to find the answer.  The average time-to-solution was 6.4 minutes, which is, all things considered, fairly long.  


What's going on?  


There are three things to keep in mind:

(1) Insider knowledge: In a sample like this you always have people who just know the answer or have some kind of "insider knowledge."  In the comments I found geologists and mining engineers who actually read the book in school.  Another person lived near the Hoover house and had picked up a lot of local knowledge by circumstance.  People with inside knowledge can often make a query that jumps right to the answer.  These are people who did queries like [ Hoover Latin text ]. *I* certainly wasn't in that group.  Maybe you were.  But for almost any search challenge I pose, there are people with some info that makes them very fast.

(2) Fast searchers:  And then there are the "fast searchers," people who picked out the most unique terms in the problem that let them get quickly to the answer.  These folks look for very "individualistic" terms (such as [ female mining engineer Stanford ] ) and manage to scan the results rapidly to find the best candidate solution.  The best searchers then double-check that candidate to see if there's a second (or third) source that confirms the insight.

(3) Slower searchers:  To be sure, some of the searchers who took between 5 and 10 minutes just got sidetracked by the material.  Kudos to them for being curious!  But I also know that 16% of the people who answered took more than 10 minutes to answer the question. 


The question that interests me is "Why did the slower searchers take so long?"  In some cases, it was a problem of making incorrect assumptions.  Paradoxically, this often happens to people who are experts in the field.  They say things to themselves like "this can't have been a 15th century text, it must have been a 17th century text... let me start there..."  


And this leads to a search lesson for today... 


Search Lesson:  Even (especially!) if you're an expert in a field, do the simple, dumb, obvious search first.  Don't dive too deeply into the content without first checking out the obvious and straightforward.  


I know this seems counterintuitive, but it's true.  (I really know this because it happens to me all the time...)  Check the basic queries first, THEN dive down.  As I wrote in an earlier blogpost, Don't miss the forest for the trees.  


Search on! 

4 comments:

  1. The question is: Why are some people fast searchers and others slow searchers in general? I and many of my friends can find just about anything on the internet, yet I know many people that can never find what they are looking for at all. Why? And I guess more interestingly, how do you make the slow searchers into fast searchers?

    ReplyDelete
  2. Tracy -- That's my job... understand why some people search the way they do, and then help them become better, more efficient, more effective searchers. Any ideas on how to do this? (I teach classes and publish content on this topic. Any OTHER ideas?)

    ReplyDelete
  3. I wonder what corollary you could draw about the people who regularly follow your search blog, (the search geeks like myself), and those who found the post via Lifehacker, and may not have otherwise seen the challenge...

    ReplyDelete
  4. As far as helping people to become better at search, I think you keyed in on it when you mentioned "individualistic terms".

    This is how I approach most of my searches, and if I find that I am getting too many stray hits in a certain non-related area, I make sure to exclude that term from my searches as well.

    ReplyDelete