Survey Results--an analysis of advice about searching

Last week we had a survey.  

It asked the SRS community about what they thought were the most important search skills.  (Take it by clicking here, if you want to see what the original survey was all about.)  

Since I do surveys professionally (for my research work at Google), I know this isn't a perfect survey.  The sample size is too small, too biased towards professional researchers, and doesn't have a broad enough set of questions to be accurate. 

I don't care.  

What I wanted from this survey is a sense of what experts think about search advice.  That is, I really wanted to hear what you, gentle reader, had to say about search methods and conducting online research.  

The good news (for me) is that many of the ideas you put into the survey are covered in my book.  The better news is that I don't cover everything you mentioned.  

To everyone who filled out the survey, many thanks for your comments.  They were thoughtful and had great insights, including a few that I hadn't thought about before!  We'll cover some of these in future posts.  


Analysis:  We got 48 responses from readers.  I was hoping for a bit more, but this is a great foundation to start.  There were also 4 questions, so that's 192 responses that I'm summarizing.  

All I did was to read through all of the responses and summarize the themes I saw.  Each of the items below was suggested by more than one person.  The rank order reflects the number of times each was written about.  #1 in the list was the most common, #2 was second most common, etc.  

Here's what you said, with a short explanation / summary afterwards. 

Most important skills:

1. Query formulation (and reformulation)

This was by far the most common skill that you think good searchers should have.  This makes sense, as the quality of your search query is a strong determiner of how successful your search will be.  Over the years I've seen that some people get stuck when they can't figure out the right search terms--that's an important barrier for getting to a successful answer.  Reformulating your query is important, both to get out of being stuck, and to hone in on what you're actually seeking.  (More on this below.) 

2. Learn from previous searches

We've talked about this in lots of SRS posts--a very important skill is that of reading and learning from your search experience.  In classes, too often I see people not reading the results page and consequently NOT learning what worked or didn't work.  Careful reading and thinking about what your search returns leads to learning, and learning leads to better searching in the future.  

3. Know what’s possible

Knowing what you can search for, and understanding the ways in which you can search, is absolutely fundamental.  In my testing of search skills, one of the things that completely blocks people from completing their search task is not knowing that you can search for X, Y, or Z (you fill in the blanks). In one dramatic example, 250 Google engineers were unable to complete a search challenge I gave them because they didn't know it was possible to search for archival aerial images on Google Earth.  (Did you?)  

4. Context

When you search for something that's more than just a simple answer, you often need to find the context that surrounds what you find.  This is especially true for complex topics such as social or historical questions.  Just reading a single web page (or, God forbid, reading only the snippets on the SERP) almost never gives you enough context information on the topic.  Read widely, and learn the when/where/why/how about your topic.

5. Lateral searching (using tabs and windows to organize your searching)

I'll be writing much more about this in weeks to come.  My colleagues Sam Wineburg and Sarah McGrew (Stanford) have written really well about this and why you'd want to do lateral searching. See: Lateral Reading (Sept, 2017).  I'll be writing about some effective strategies for doing this.  

6. How to limit your searches to a particular domain 

Limiting the scope of your searching is simple, if you know how to use site: or filetype: -- those are the simplest ways to get results with certain kinds of properties.  You use site: when you want results from a particular web site or organization.  Thus, [site:CDC.gov influenza] will give you results with a distinctly CDC perspective.  Likewise, you can exclude certain sites with the minus operator.  [-site:CDC.gov influenza ] Or you can search multiple sites or entire domains:  [site:.IN Brexit OR site:.BE Brexit] will search for "Brexit" on Indian or Belgian sites. 

7. Ability to evaluate what you find

Yes!  This is critical... and requires much more discussion than I have room for here.  We'll return to evaluation rules-of-thumb in future blog posts. 

8. How to find experts and learn what they know

This is a great way to find advanced, expert content on a topic.  The trick here is to locate experts that are truly expert (why do you think they're an expert?) and then tracking down their writing.  The other trick here is to always locate more than one expert.  Ideally, you want to see the varieties of thought on a topic (there could be more than one point-of-view... in some cases, there can be 4 or 5!), and get experts from each perspective.  

9. Critical thinking 

A critical thinker is always wondering how could this be wrong?  And how can I break this big problem down into pieces? There are many approaches to critical thinking, but these two heuristics work well for me.  A critical thinker criticizes an idea, some writing, or a point-of-view... but does so productively, not to just be mean-spirited about it.  Critical analysis is trying to break something into its components to understand how it works.  It does not assume that authorities are always correct, but calls things into question.  (This is why critical thinking annoys some people.)  Of course, you want to do your critical thinking with care, and not be annoying about it.  But it's an essential skill.  

Most important attitudes:

1. Persistence

Persistence in the face of failure is the #1 attitude (or trait) that was mentioned by those surveyed.  I agree.  It's obvious in my search classes when someone has this attitude, and when they don't.  Much of what I teach is attitude encouragement.  "If that didn't work, what do you think would work?"  And "Just try one more search.."  

2. Curiosity 

You'd think this would be obvious, but I agree.  Intrinsic curiosity about the world is essential.  On the other hand, I don't know how to communicate this other than by being curious myself and showing what kinds of things you can learn about the world through satisfying your curiosity.  (In many ways, my book is exactly this--ways to find answers to your questions about curious things.)  

3. Adaptability 

Adaptability is just the ability to respond to changing circumstances.  In online research, it's the ability to act effectively when things go wrong (e.g., getting 404 errors, finding that your favorite website no longer supports that service, etc.)  You'll also find that you need to adapt to changing tools (new features introduced, old ones go away) and data.  

4. Enthusiastic

 Someone's enthusiasm goes a long way in making both persistence and curiosity work.  I've seen people who were not great searchers manage to succeed because they had great enthusiasm for their topic, which in turn made them persistent, curious, and adaptable.  (These people are always great to work with.  Their enthusiasm for the topic means they want to learn how to be better and more accurate.)  

How to ask good questions:  

These answers were pretty interesting, and all over the map.  Here's my condensation of the top 10 ideas. 

1. Be specific / be clear about what you’re asking

2. Ask yourself:  WWDD  (what would Dan do?) 

I was surprised about how many people said this.  Aww.. gee... thanks! 

3. Who would know the answer to your question… how would they put that info online?

This is a variation on the "find an expert" heuristic from above.  But I like it as a heuristic for asking questions too.  How would an expert think about your question?  

4. Think of alternative ways to ask the question

5. Start large – don’t zero in on a specific topic

6. When stuck, work from specific examples

7. Always ask why? 

That is, ask why about the answers you’re finding.  Why is this true?  And How do you know?  This almost always leads to a better question. 

8. “Predict the answer”

One way to create a question is to think forward to what an answer would look like.  That is, if the answer looks like this, what question would let me get to that?

9. Ask yourself WHY do you want to ask this question.

This is a classic library reference desk question.  A great reference person always tries to dig into what you want to know.. and why.  Knowing the answer to why tells you a lot about how to frame your question.  

10. Be open to nuance. 

Don't give up on the subtlties of a question.  You can almost always ask "Is there more I should know?"  That's a useful question in almost all cases. 

Other advice from the survey

1. be aware of other search tools (e.g., thesaurus, dictionaries, geographical tools, …)

2. don’t limit yourself to English and US sites.

3. search for tools to help you accomplish your task

4. recognize that Google doesn’t have everything

5. be humble in your searching

6. consider the motivation for whoever is writing whatever you’re reading. Why did they write this?  Why did they post it?

7. be aware of confirmation bias

Summary 

There's obviously a lot that's been packed into these answers.  I'll be unpacking some of these themes over time.  

You'll be amused to know that the attitudes section is almost exactly what I cover in chapter 19 of my book.  Thanks for confirming my intuitions!  

See you next week when we discuss the answer to the mysterious avocado seed question! 

Search on.