Thursday, June 7, 2018

Survey Results--an analysis of advice about searching

Last week we had a survey.  

It asked the SRS community about what they thought were the most important search skills.  (Take it by clicking here, if you want to see what the original survey was all about.)  

Since I do surveys professionally (for my research work at Google), I know this isn't a perfect survey.  The sample size is too small, too biased towards professional researchers, and doesn't have a broad enough set of questions to be accurate. 

I don't care.  

What I wanted from this survey is a sense of what experts think about search advice.  That is, I really wanted to hear what you, gentle reader, had to say about search methods and conducting online research.  

The good news (for me) is that many of the ideas you put into the survey are covered in my book.  The better news is that I don't cover everything you mentioned.  

To everyone who filled out the survey, many thanks for your comments.  They were thoughtful and had great insights, including a few that I hadn't thought about before!  We'll cover some of these in future posts.  

Analysis:  We got 48 responses from readers.  I was hoping for a bit more, but this is a great foundation to start.  There were also 4 questions, so that's 192 responses that I'm summarizing.  

All I did was to read through all of the responses and summarize the themes I saw.  Each of the items below was suggested by more than one person.  The rank order reflects the number of times each was written about.  #1 in the list was the most common, #2 was second most common, etc.  

Here's what you said, with a short explanation / summary afterwards. 

Most important skills:

1. Query formulation (and reformulation)
This was by far the most common skill that you think good searchers should have.  This makes sense, as the quality of your search query is a strong determiner of how successful your search will be.  Over the years I've seen that some people get stuck when they can't figure out the right search terms--that's an important barrier for getting to a successful answer.  Reformulating your query is important, both to get out of being stuck, and to hone in on what you're actually seeking.  (More on this below.) 
2. Learn from previous searches
We've talked about this in lots of SRS posts--a very important skill is that of reading and learning from your search experience.  In classes, too often I see people not reading the results page and consequently NOT learning what worked or didn't work.  Careful reading and thinking about what your search returns leads to learning, and learning leads to better searching in the future.  

3. Know what’s possible
Knowing what you can search for, and understanding the ways in which you can search, is absolutely fundamental.  In my testing of search skills, one of the things that completely blocks people from completing their search task is not knowing that you can search for X, Y, or Z (you fill in the blanks). In one dramatic example, 250 Google engineers were unable to complete a search challenge I gave them because they didn't know it was possible to search for archival aerial images on Google Earth.  (Did you?)  

4. Context
When you search for something that's more than just a simple answer, you often need to find the context that surrounds what you find.  This is especially true for complex topics such as social or historical questions.  Just reading a single web page (or, God forbid, reading only the snippets on the SERP) almost never gives you enough context information on the topic.  Read widely, and learn the when/where/why/how about your topic.

5. Lateral searching (using tabs and windows to organize your searching)
I'll be writing much more about this in weeks to come.  My colleagues Sam Wineburg and Sarah McGrew (Stanford) have written really well about this and why you'd want to do lateral searching. See: Lateral Reading (Sept, 2017).  I'll be writing about some effective strategies for doing this.  

6. How to limit your searches to a particular domain 
Limiting the scope of your searching is simple, if you know how to use site: or filetype: -- those are the simplest ways to get results with certain kinds of properties.  You use site: when you want results from a particular web site or organization.  Thus, [ influenza] will give you results with a distinctly CDC perspective.  Likewise, you can exclude certain sites with the minus operator.  [ influenza ] Or you can search multiple sites or entire domains:  [site:.IN Brexit OR site:.BE Brexit] will search for "Brexit" on Indian or Belgian sites. 

7. Ability to evaluate what you find
Yes!  This is critical... and requires much more discussion than I have room for here.  We'll return to evaluation rules-of-thumb in future blog posts. 

8. How to find experts and learn what they know
This is a great way to find advanced, expert content on a topic.  The trick here is to locate experts that are truly expert (why do you think they're an expert?) and then tracking down their writing.  The other trick here is to always locate more than one expert.  Ideally, you want to see the varieties of thought on a topic (there could be more than one point-of-view... in some cases, there can be 4 or 5!), and get experts from each perspective.  

9. Critical thinking 
A critical thinker is always wondering how could this be wrong?  And how can I break this big problem down into pieces? There are many approaches to critical thinking, but these two heuristics work well for me.  A critical thinker criticizes an idea, some writing, or a point-of-view... but does so productively, not to just be mean-spirited about it.  Critical analysis is trying to break something into its components to understand how it works.  It does not assume that authorities are always correct, but calls things into question.  (This is why critical thinking annoys some people.)  Of course, you want to do your critical thinking with care, and not be annoying about it.  But it's an essential skill.  

Most important attitudes:

1. Persistence

Persistence in the face of failure is the #1 attitude (or trait) that was mentioned by those surveyed.  I agree.  It's obvious in my search classes when someone has this attitude, and when they don't.  Much of what I teach is attitude encouragement.  "If that didn't work, what do you think would work?"  And "Just try one more search.."  

2. Curiosity 

You'd think this would be obvious, but I agree.  Intrinsic curiosity about the world is essential.  On the other hand, I don't know how to communicate this other than by being curious myself and showing what kinds of things you can learn about the world through satisfying your curiosity.  (In many ways, my book is exactly this--ways to find answers to your questions about curious things.)  

3. Adaptability 

Adaptability is just the ability to respond to changing circumstances.  In online research, it's the ability to act effectively when things go wrong (e.g., getting 404 errors, finding that your favorite website no longer supports that service, etc.)  You'll also find that you need to adapt to changing tools (new features introduced, old ones go away) and data.  

4. Enthusiastic

 Someone's enthusiasm goes a long way in making both persistence and curiosity work.  I've seen people who were not great searchers manage to succeed because they had great enthusiasm for their topic, which in turn made them persistent, curious, and adaptable.  (These people are always great to work with.  Their enthusiasm for the topic means they want to learn how to be better and more accurate.)  

How to ask good questions:  

These answers were pretty interesting, and all over the map.  Here's my condensation of the top 10 ideas. 

1. Be specific / be clear about what you’re asking

2. Ask yourself:  WWDD  (what would Dan do?) 
I was surprised about how many people said this.  Aww.. gee... thanks! 

3. Who would know the answer to your question… how would they put that info online?
This is a variation on the "find an expert" heuristic from above.  But I like it as a heuristic for asking questions too.  How would an expert think about your question?  

4. Think of alternative ways to ask the question

5. Start large – don’t zero in on a specific topic

6. When stuck, work from specific examples

7. Always ask why? 
That is, ask why about the answers you’re finding.  Why is this true?  And How do you know?  This almost always leads to a better question. 

8. “Predict the answer”
One way to create a question is to think forward to what an answer would look like.  That is, if the answer looks like this, what question would let me get to that?

9. Ask yourself WHY do you want to ask this question.
This is a classic library reference desk question.  A great reference person always tries to dig into what you want to know.. and why.  Knowing the answer to why tells you a lot about how to frame your question.  

10. Be open to nuance. 
Don't give up on the subtlties of a question.  You can almost always ask "Is there more I should know?"  That's a useful question in almost all cases. 

Other advice from the survey

1. be aware of other search tools (e.g., thesaurus, dictionaries, geographical tools, …)

2. don’t limit yourself to English and US sites.

3. search for tools to help you accomplish your task

4. recognize that Google doesn’t have everything

5. be humble in your searching

6. consider the motivation for whoever is writing whatever you’re reading. Why did they write this?  Why did they post it?

7. be aware of confirmation bias


There's obviously a lot that's been packed into these answers.  I'll be unpacking some of these themes over time.  

You'll be amused to know that the attitudes section is almost exactly what I cover in chapter 19 of my book.  Thanks for confirming my intuitions!  

See you next week when we discuss the answer to the mysterious avocado seed question! 

Search on.  


  1. Curious results Dan. Nicely done. On experts, here are 3 that I worked on recently.
    1. A woody thing I found on a beach. I did all the online stuff and got nowhere. I took it to our Big Royal British Columbia Museum where 8 people spent an hour and a half examining and discussing it before giving me an answer.
    2. An ancient pal recounted a curious story from WW2 when his English ship was threatened by a German ship but no shots were fired. I puzzled over this several years and finally asked the Royal Navy Forum. Didn't take long to find it was the Spanish flagship not a German ship.
    3. I have an old photo from about 1920's in which my grandfather is standing near a Rolls-Royce. Was it his? What year was this pix taken. I asked the editor of a RR magazine for help. He made guesses then sent my query to the world expert who promptly replied with all details: coachmaker, serial number, size of engine, later pix and best of all--it still exists in pristine condition and in the world's largest private RR collection which has dandy big pix of it.
    4. Washing machine would not spin. YouTube etc no help. A phone call later the expert came and removed a sock that was jamming the works.

    I love experts. Like you. And the birders who can tell me what I see or hear

    1. re: #3 – was your Grandfather Captain Sir Malcolm Campbell, MBE?
      at Brooklands
      originally owned by Captain Sir Malcolm Campbell
      off pinterest
      btw - was it the Spanish battleship Alfonso XIII? …curious about the beach find…

    2. remmij. My grandfather not Campbell.
      This is the very car 50 yrs after my photo was snapped. Cheers jon who did see the citrine wagtail--at least they the xperts said I did.

  2. The Spanish ship they think can only be the heavy cruiser Canarius. The curious beach find was a rounded piece of shelf fungus.

  3. Even with just a small pool of 48 responses, it's good to see that people are attempting to learn from their previous mistakes when searching. I spend plenty of time doing research in my work, and the last thing I want to do is waste time looking in the wrong corner of the internet for useful information. Knowing how to search is a great way to stop wasting time in the workplace, so you can get your work done efficiently and on time. Being specific is a great way to save time that would otherwise be wasted on useless information!

  4. Anne and I always enjoy the searches. We have learned so much over the years!