Tuesday, November 22, 2011

Intuition and counterintuition


Intuition:  What’s the deal with intuition these days?  It seems to be on everyone’s mind, a brainworm on the loose.  People are claiming that tablets like the iPad “tap into intuition,” that Steve Jobs had an “intuitive designer’s sense,” George Bush “trusted his gut feelings” about the presence of WMD in Iraq, Kim Kardashian decided that “intuition led me to divorce” after 72 days of wedded bliss,  and the Huffington Post writes “Science says to trust your gut.”  Intuition seems to be more valued than ever, although there seems to have been no recent upgrade in our collective intuitive skills.  Is intuition really just the flowering of some inner, secret power? 

By contrast, the current movie “Moneyball” is about the success of the Oakland A’s team of 2002, a team that was put together with guidance of  some clever statistical analysis by their general manager, a baseball quant-jock if there ever was one.   Even Roger Ebert says the film is about “the war between intuition and statistics.”  Is there really a war going on? 

In his book “Blink,” Malcolm Gladwell talks about intuition as “thin-slicing” experience based on training up your experience base after 10,000  hours of practice in a field.  If you have that much time-on-task, practicing and learning how to operate efficiently in a field, then you’re capable of rapidly assessing a situation based on few, rapidly scanned clues, and somehow coming up with a quick recognition of what’s going on. 

Here's the thing to know:  How much time have you spent doing search?  My best estimate is that I've done about 5,000 hours of search since I first started using Google in late 1998.  That is, I started practicing my search skills around 13 years (or 4748 days) ago.  

If I've done a bit over 1 hour of search / day since then (which seems reasonable), that means I've invested ~5,000 hours of practice.  While that's a lot of time, it's only half of the 10,000 hours that are usually needed for real expertise. 

System 1, System 2: Nobel Prize winner Daniel Kahneman points out that people have two different and parallel systems of thought when they confront  problems.  The cleverly named “System 1” is a fast recognizer of situations in context—it identifies and labels objects, picks up on relationships, and does so by recognition, rapidly, rather than by deliberation.  

Then there’s “System 2,” the slower, more deliberate, symbol-pushing and rational part of our minds. 

As an example, System 1 recognizes that the number sequence 2, 4, 8, 16, 32… is just doubling from one to the next.  It would be System 2 that lets you realize that this is also the powers of 2.  Of course, if you’re a computer scientist, the powers of 2 has become, over many repeated exposures, something that’s a System 1 effect.  For non-CS majors the problem 2 * 256 is a System 2 task.  For CS-majors, it’s a System 1 task—you recognize the pattern and say, “it’s 512…” without thinking much.  In this sense, intuition is what you’ve been trained to expect to perceive.  It is the power of repeated exposure and the accumulation of inarticulate recognition skills.  

About System 1 Kahneman writes in his book Thinking, Fast and Slow
“We are prone to think that the world is more regular and predictable than it really is, because our memory automatically and continuously maintains a story about what is going on, and because the rules of memory tend to make that story as coherent as possible and to suppress alternatives. Fast thinking is not prone to doubt.”

Nor is System 1 particularly good at noticing contradictions.  We swim in a sea of counterintuitions--things that seem to be intuitively correct, but are not.  We have difficulty seeing the sea we swim in simply because we swim in it all the time.  

Some examples:  
  • How is it that clouds made of water vapor yet can float mid-air?  (Water is awfully heavy.)  
  • The world is visibly and obviously flat—yet we now believe that it’s intuitively obvious that the world is round.  (Trust me, historically speaking, that wasn’t obvious at all!)  
  • When we’re on a merry-go-round, our intuition tells us that the force is outward—that centrifugal force is really trying to thrust us radially away from the center, not on a tangent along the direction of travel.  
  • Dense things are typically opaque, except for glass and water, which are “intuitively obvious” exceptions to the rule.  

Counterintuitive: Except that they’re not obvious, it’s just the pattern you’ve seen so often that System 1 doesn’t even pick up on the contradiction. 

Intuitive thinking is primarily what you’ve experienced and on patterns you pick up.  While it’s fast, it’s also errorful.  Dan Ariely has asked hundreds of Princeton undergraduates the following question:  “A bat and ball together cost  $1.10.  The bat costs $1 more than the ball.  How much is the ball?”  It’s a simple question, but around 50% of Princeton seniors get it wrong and say that the bat costs 0.10.  (That can’t make sense.  If the ball is 0.10, then “$1 more than the ball” means that the bat is $1.10 – adding the bat and ball prices together means the bat + ball are $1.20.) 

So, why do they get it wrong so often?  

Because they don’t check their work.  Why?  Because it’s a bit of a hassle, and the answer is so obvious and apparent (and the stakes are so low) that it’s not worth the effort.  This is characteristic of many intuitive answers—it’s so *obvious* that it’s not worth checking. 

Seems to me that there’s not really a contradiction between System1 (intuition) and System 2 (rational) modes of knowing.  Instead, one’s just faster than the other.  Both are useful, both are necessary.  But the great achievement of science has been to tell us that intuition always has to be checked.  In essence, science is the overturning of intuition by slow, painful, careful System 2 reasoning.  Clouds float because cloud-borne water droplets are really, really tiny and are kept aloft by random air movements.

A note from Dan’s System 2:  Cloud droplets average between 1 and 100 microns.  A typical droplet 20 microns in diameter is 4.2 picoliters in volume with a weight of 4.2 nanograms, falling at a terminal velocity of 0.02 mph.  Thus, an updraft at just over 0.02 mph will keep the cloud in the sky.  That’s not much updraft.  Or to look at it another way, if the cloud formed at 10,000 feet, at 0.02 mph it will take nearly 4 days to fall to the ground.

The trick is to know when to use one mode versus the other.  “Trust the force, Luke” Obi-Wan whispers into our ears.  But the same time, you can’t build a pan-galactic empire with massive infrastructures based on your intuitions about mechanical engineering, power grid design, or waste management systems. 

We know there are a number of rather rapid inferences that are System 1.  A while ago Bob Zajonc showed that determining preference is much faster than cognition… that is, it anticipates cognition.  You see this when making choices about which kind of thing you prefer over something else—be it mates, brands of peanut butter or random Chinese characters.  As he said, “Preferences need  no inferences.” It’s fast and gets there before you’ve figured out what you really should want.   That’s what System 2 reasoning is for. 

It strikes me that we have a lot of brainware dedicated to specific fast System 1 type tasks, and even those mechanisms get it wrong much of the time.  We’ve all seen various visual illusions that we KNOW can’t be true—a vertical table that looks much longer than a horizontal table; two patches of color that look very different in a scene, but can be easily shown to be the same color when placed side-by-side. 

The strange thing is that visual illusions persist, even when you know they’re wrong; even when you’ve measure the differences; even when you put the color patches side by side.  Even though you do the measurements yourself, you can’t convince your vision system to accept your measurement as truth.  It’s as though you haven’t learned anything in the last minute of experience.  

(In the example below, squares A and B are the same color.  Use an eyedropper tool to measure the colors and find they're the same. This illusion image is from Wikipedia,  created by Edward H. Adelson, Professor of Vision Science at MIT in 1995.)  



Even though we DO more vision than anything else we do with our brains,  we still have predictable, repeatable mistakes in vision.  How can we be better in domains where we don’t have so much hardwired brain? 

This is the realm of the cognitive illusion, and is even more problematic.  We know that if you set up a program with an opt-out vs. an opt-in choice, the differences in selection rate are huge.  In one European study, the rate of people selecting to be in a organ donation program was massively different: in Sweden the participation rate is 86%, while in Denmark the participation rate is only 4%.  The difference is that one is opt-in while the other is opt-out. 


Dan Ariely points out that when we’re filling out the form we 

“feel as though we’re making decisions… but the person who designs the form that is actually making the decision for you.  You like to think that the options don’t influence us, but that’s not true.”  

The illusion is that you have control over your destiny.  That is, you think your System 2 rational decision-making system will actually express your deeply-held, fundamental beliefs.  “I believe that organ donation is a social good, so I’ll participate.”  That’s the story you might tell yourself.  But statistically speaking, the choice is made for you when the form is designed as opt-out. 

We don’t recognize the illusions we all live with day-to-day, we’ve come to accept them as intuitive and obvious.  Normal is what you’ve grown up with, the non-intuitive, confusing, difficult  high-technology world is whatever was invented after you turned 8, and stopped seeing everything as pre-destined.  

Isn't that intuitive?  


Friday, November 18, 2011

Verbatim mode - Google without interpretation

Remember the ruckus about the + operator going away?


We knew about the issue, and this week Google has announced a new search mode that you might want to know about. 


Verbatim mode was announced on the Inside Search blog (a blog I recommend you read, actually, since most of the interesting announcements about new Google search features will appear there first).  


What's Verbatim search?  It's Google search without any synonymization, spell-correction, personalization or other interpretation.  That is, it's just a basic search without any alterations in what you typed into the query box.  To be clear, 99% of the time, those alterations actually improve your search results quite a bit.  But every so often you really want just what you typed.  Verbatim mode is for those situations.  


To turn on Verbatim, you first open the "More search tools" button on the left hand panel... 

That is, click on the bottom option, just below "Custom range..."    This will open up a new set of options below that point.  It should look like this: 

When you click on Verbatim, you'll put your subsequent searching into this hyper-literal mode.  


As we've discussed before, sometimes a search term seems so obviously misspelled (to Google) that it can't resist spell-correcting it for you.  


 But as you know, in this case we REALLY want to word to be unaffected by spell-correction or synonymization.  You already know that you can double-quote the word like this: 
 ... to get the Finnish folk music style that you were really searching for.  


Now, when would you use Verbatim?  
Whenever you want to do longer queries that you don't want altered at all.  
For instance, a multi-word query such as [ joiker music finland ], a trip to Verbatim mode might be exactly what you want.  Here's the query in default search mode: 
But you might want to search this in Verbatim, as below.  The first result is the same, but the next several results are somewhat different.  
Notice the blue bar at the top of the results.. that's how you know you're still in Verbatim mode.


The difference is probably easiest to see in a side-by-side like this.  Here I'm searching for "gyros":  
As you can see, the Verbatim mode search (on the right) gives a very different answer--one that's NOT localized or personalized.  


Hope you find this new tool useful!  Let me know how you like it!  


Search on. 

Thursday, November 17, 2011

Answer: Where's the quarry?

The fast answer to the question:  the Tennessee pink marble for the Newseum First Amendment tablet comes from the Endsley quarry, which is in Friendsville, TN at  35.75413, -84.1334


Here's what I did to find it.  


I did a straight-forward search on [ Newseum stone ] which led me to this article in KnoxvilleBiz.com 

"...The marble came from the Endsley Quarry in Friendsville and was provided by Tennessee Valley Marble before the company was purchased last year by Tennessee Marble Co.  
Architects chose Tennessee pink marble for the tablet because the same stone was used on the exterior of the National Gallery of Art, which is across the street, and in several other prominent buildings and monuments throughout the city..."  

Wow.  THAT was easier than I thought.  

I did a big of digging into the Tennessee Valley Marble company and found that it actually runs several quarries, but that the Endsley quarry, just outside of Friendsville, wasn't used for many years, but was re-opened for the Newseum project and is still active.  


By using the "Drop lat/long marker" tool in Google Maps, it was pretty simple to find out the lat/long. 

Along the way I found that they needed only 50 tons of stone for the tablet, but in order to get enough decent stone, they had to quarry out (and discard?) around 278 tons.  

If you look to the right (east) of the quarry, you can see a stone cutting facility that you can't see from the road.  But if you zoom in, you can spot large blocks of stone sitting around outside, looking like small cars. 



But if you look closely and use the measuring scale in the lower left of the image, all of the blocks at the quarry are actually fairly large.  They're not at "finished size."  Most of the blocks in this image above are around 10 feet on a side.   
It looks to me as though they're quarried here, cut in to large blocks, then shipped off to the main cutting facility in town at the Tennessee Marble Co, on West Vinegar Valley Rd.  (I found this location by zooming the map out a bit and doing a query for [ Tennesee Marble Company ].)  

At this location a few miles away from the quarry, you can spot many more slabs that are of the right size to be sent off for final engraving and installation.  You can watch the video of the engraving and installation by the Rugo Company which did all of the work of actually putting words onto the stone.  In this video you can see that the individual slabs are really fairly small--the kind of slabs that would be precision-cut in a place like the one above.  

As several of you geologically inclined readers pointed out, Tennessee Pink marble actually isn't marble, but a kind of crystalline limestone that looks a lot like marble when polished.  As the Wikipedia article on marble points out: "Geologists use the term "marble" to refer to metamorphosed limestone; however stonemasons use the term more broadly to encompass unmetamorphosed limestone."  

So Tennessee pink is really just unripe marble.  It needs a bit more metamorphosis to mature from limestone into marble!  

All the comments this time were great!  Many readers figured this one out.  Excellent.  We are getting smarter! 

Search on! 


Wednesday, November 16, 2011

Wednesday search challenge (Nov 16, 2011): Where's the quarry?

I'm in Maryland this week and had the chance to visit the Newseum in downtown Washington DC.  It's an incredible museum of news, a place I highly recommend when you visit the DC.  It has a great view of the capital building and on the top floor, a wonderful vista of the entire mall.  


While I was there I spoke with man who told me that the beginning 45 words of the First Amendment to the constitution are engraved on giant slabs of marble on the front of the building.  He went on to say that it was made of the same stone as was used to build the museum directly across the street.   "The idea was that they wanted it to match, so they went and re-opened the original quarry."  


He didn't know where the quarry was that was used as the source of stone, but he was pretty sure it was "down south" somewhere.  


I was curious about where such a massive chunk of rock would come from, so I searched it out.  It took me about 3 minutes to find an aerial photo of the quarry from whence it came.  


I now know where the quarry is.  Can you find that aerial image and provide the lat/long of the quarry?  


For extra credit, can you find the lat/long of the place where the stone was cut to shape? 


Search on!

Friday, November 11, 2011

Answer: How do people spend their time?



Whaddya know?  This turned out to be simpler than what I thought--for an unexpected reason.  


Here's what I did.  My first query was to see if I could find any database that had reasonable information about how Americans spend their time day-to-day.  


Americans spend time ] 


This query led me quickly to the American Time Use Survey at the US Bureau of Labor Statistics (BLS).  This is their job: collect data and stats on American behavior across many different dimensions.  


 I drilled down almost immediately to their data sets about time use and found data from their big survey of ~13,200 Americans in 2010.  The metadata for this survey is here (which is where they write down all the details of how the surveys were coded, the survey question form (ever want to see the script that surveyers use when asking you questions--this is it), etc.  


Eventually, I got to the summary table of "Time spent in primary activities (1) and percent of the civilian population engaging in each activity, averages per day by sex, 2010 annual averages."  I exported that data into my spreadsheet, and generated the following chart rather quickly.  



Whenever you look at a chart like this, questions and insights immediately spring to mind.  For instance, WHY is the average amount of work / day only 4.09 for men and 2.94 for women?  (Or 3.5 average between the two.)  Or, to answer my original question--"Do I spend a lot of time in email?"  The answer is "yes, you do... but you're in Silicon Valley, what did you expect??"  


You might prefer to see the data in this format:  




(Same data, just different chart.)


Remember that this is a sample of  people nation-wide, balanced across demographic categories (age, gender, location, etc.) and included a representative sample of unemployed people as well.  Keep in mind, if you work a standard 40 hour work-week for 50 weeks of the year, you're really only working 4.3 hours/day averaged over the entire year.   


Contrast that with the number of hours / day if you're engaged in that activity--7.82, which reflects more what you'd expect for number-of-hours worked given that you're working on that day.  


We could continue analyzing the data, which is really interesting, but I want to return to the search question for a minute.  


Why did I say this was simpler than I thought? 


Answer:  Because I had a particular solution path in mind when I started out.  I would (1) locate the data, (2) extract it, and (3) analyze it.  

I never expected that the BLS would have already done this for me!   As gasstationswithoutpumps commented,
     "I searched for [ hours spent american statistics ] which got me to
     http://www.bls.gov/news.release/atus.nr0.htm which has pointers
     to tables for 2010 statistics divided in lots of ways. (in particular...) 
    Table 12 http://www.bls.gov/news.release/atus.t12.htm 
     seems to be the one you want, with sleeping and watching
     TV as the two biggest categories."

I never expected that they would have already done the analysis, with charts and everything!  (Note that this chart from BLS is slightly different than mine--they're charting the time use on an average workday for employed persons with children.  The chart above is for everyone in the sample set.)  



Now....

On the other hand I read analyses of how people spend their time as compiled by various other organizations.  Neilsen, for instance reports that people spend 4.9 hours/day watching television.  That's very different than the amount that the BLS finds in their data.   You can find similar differences for other measures as well (e.g., internet use; amount of time spent care-giving; household activities; ....).  

The big takeaway is that you have to be VERY careful that you know exactly what data is going into your analysis.  It matters how it was collected, what the coding scheme was, and how the sample was chosen.  Was it random digit dialing (as in the case of the BLS data), or was it "Neilsen Family" people?  The outcome can vary tremendously. 

One of the beautiful things about being able to find data of this kind is the ability for us to download and analyze it according to our own interests.  The folks at FlowingMedia did a beautiful interactive version of this same data by building on D3, Mike Bostock's visualization system.  The image below is just a single frame from their system.  You should visit the FlowingMedia interactive visualization (click on the demographic segments in the upper right) to get the full experience.  



Bottom line:  Searching for a data that you can easily download, understand and do your own analysis of is a great feature of the new Data Age.  You should learn how to get to data AND how to understand it.  That metadata stuff turns out to be deeply important.  

Yours in finding ever more data! 

Search on! 



Thursday, November 10, 2011

Search Challenge (November 9, 2011): How do people spend their time?

It might not look like much, but this is a compelling graphic: 
It's a chart of how people in the US spend their time during an average day. 


Can you figure out how people spend their time?  You can probably guess what the big orange and big blue segments are, but what makes up the other 30% of the day?  


This question came about from a simple discussion at work the other day.  I know that I spend about 2.5 hours each day just doing my email.  Was that normal?  Is it a high number?  A low number?  


So if you take the US as a whole (statistically speaking), can you figure out how normal people spend each day? 


The best solution here is to be able to create a chart something like this one (although I'll show a different graphic tomorrow).  You need to find data you believe--a source that's credible, or at least one that uses some kind of data-gathering mechanism that you believe.


Be sure to tell us how you found your data and why you believe it.   


Search on! 



Tuesday, November 8, 2011

Alexis Madrigal's research process

In his recent Google+ post, the Atlantic Monthly writer Alexis Madrigal noted down a recent research task that he did.  


I'm copying most of the text of his post here so I can comment on bits of it... 


He wanted to do some background research on "how people want to tow icebergs for fresh water supply."  (My quick characterization of what he wanted to find out.)  


What he did to research this topic was pretty interesting, and since he wrote it down we can take a look at his process.  

From his post: "... I knew that "iceberg towing" was a unique phrase that could be pumped through every search engine imaginable to find interesting information. And that's really what enabled me to write a comprehensiveish post about the phenomenon in a short amount of time. Here was my rough order of operations, although given tabbed browsing, many of these things were going on simultaneously..."  

Madrigal then goes on to list the things he checked.  



He's right to notice that a phrase like "iceberg towing" is so perfectly descriptive of what he wants (and wonderfully uncommon) that almost anything he finds is going to be just about on-target.  


My comments come after each of his steps.  

1. Google Books (first with only free books shown, then later with all previewable books shown)



This is actually a great first move, as Google Books has not just books, but also many popular magazines.  Popular Mechanics (1978, vol 149, no 1, pg 47) has a wonderful picture of a ship nearly being sunk by shards of ice falling from the iceberg.  That's one of the values added by popular magazines--images and "inside stories" that are often missed by more academic monographs and technical articles.  

2. Standard Google search, followed by searches just within .gov sites and then looking only for PDFs



Using searches like [ "iceberg towing" filetype:pdf ] and [ "iceberg towing" site:.gov ].  These limit the search results to PDFs (usually articles from academic sources) and to government sites (.gov).  Another useful trick might have been to try [ "iceberg towing" site:.edu ] 

3. The Chronicling America collection, late 1800s and early 1900s newspaper accounts as well as the other Library of Congress databases, though almost all of them came up empty. 



I have admit I'm not surprised by the lack of content here.  Still, it's a smart move to check these kinds of special collecitons, and it doesn't take long to check.  

4. YouTube



I think I would have checked on Google Video, as it's a superset of the YouTube results.  One search would do both sets of content.  But Alexis is following advice I give in all of my classes about being a good searcher:  Check other kinds of content beyond just web-search.  

5. Flickr Commons, then Flickr



This seems redundant as the images returned from both sets are the same.  On the other hand, you can quickly figure out that perry709 was the person who took those photos, and by doing the obvious searches, you can find out that he even has an iceberg tattoo and clearly works in marine research.  But we digress.  My point is that a little bit of chaining from a known source can lead to even more discoveries and confirmation about the credibility of a source.  (And I found a few photos that perry7009 hadn't put onto Flickr, but were under his Imgur account--images that hadn't appeared anyplace else.)    

6. Here, I pinged Twitter, which brought Project Habakkuk and Brewster's Millions to my attention. Twitter is great for testing what people first think of when they hear some phrase like "iceberg towing." It lets you play defense against common assumptions.



I completely agree.  Even though I work at Google, one of the great things I've seen in truly power searchers (what I call "ultra searchers") is that they always mine their social networks.  Twitter is a great, fast way to tap into a large community, including people you might not know directly, but who can point you to places you would have never found in any other way.    

7. Google Scholar, followed by searching in the footnotes of the papers found there, which led to the discovery that there were two major papers put out in 1973, one in the Journal of Glaciology by Weeks and Campbell (later published in modified form in The Bulletin of Atomic Scientists) and another by RAND researchers J.L. Hult and O.C. Ostrander. Both were gold mines of information.



Google Scholar is one of the great, but underused, repositories around.  For a research topic like this that skirts around in the academic literature, this is a great place to go.    


8. By now, many signs pointed to John Isaacs at Scripps Oceanographic Institute as the key figure in the midcentury, so I went to Scripps' archives and found a bunch of photos of the guy as well as a book about him, which was previewable in Google Books


Within a few clicks you'll arrive at the Scripps Archive, which is (apparently) not indexed by Google!  Discovering dark-web repositories (as is often the case with special-library archives that run their own search engines of their content) is often a way to find things that other, less motivated searchers, will never discover.  If you're looking for a new story or a scoop, this is a great technique for finding good stuff.     


The other thing he could have done was to use the UCSD "Ask a Librarian" service.  Many libraries run a special email (or live chat) answer service.  I think I might have checked out the "Ask a Librarian" about what resources Scripps has on "iceberg towing" -- you never know what's not in any online archive and just sitting in an archive box on the librarian's desk.  

9. Meanwhile, the RAND report led me to a slough of other sources and filled in the deep history of iceberg towing from South America.  



Makes sense to me too. 

10. I decided to follow up on an early hit from TIME Magazine's online archives about a conference sponsored by Prince Faisal of Saudi Arabia at Iowa State University (!) about iceberg towing. Turns out, Iowa State maintains a small archival collection about the conferences, which began in 1979 and ran for a few years.



This is a fantastic find.  (Check out the site: http://www.lib.iastate.edu/arch/rgrp/0-4-4.html)  I agree with Alexis, this was pretty much unexpected, so it's another great resource to draw on.  

11. That was pretty much it. I did some more extensive searching around "iceberg towing + Labrador Sea" and that yielded a bunch of interesting information about the business and science of towing bergs for the oil industry. Specifically, a 2003 Smithsonian mag profile surfaced and an excellent document from Grand Banks Iceberg Management from 2007, which pretty much tells you anything you need to know about how to actually get the practice done.


Not to be picky, but a search like [ "iceberg towing + Labrador Sea" ] won't actually find you much.  (The + was ignored, and so the query is really just [ "iceberg towing Labrador Sea" ] -- which is okay, it finds a few results.  



But it makes me think that a REALLY interesting search might be to use this strategy more generally.  Example:  use the "fill in the blank" (aka "wildcard") mechanism.  Like this: 


[ "iceberg towing" "* sea" ]  -- which finds a video I hadn't seen before 


or... 


[ "iceberg towing"  "* ocean" ] -- which finds even more results


The strategy there is to try and find new kinds of specific results (e.g., "Arctic Ocean") rather than just the Labrador Sea.  

All in all, I was pretty impressed--Alexis showed a real mastery of the art of search.  Kudos to him for documenting a longish, deep and determined search strategy and sharing it with everyone on the web. 



Search on, Alexis! 






Monday, November 7, 2011

Where's the Advanced Search button? (It's where the Language Tools went)

As you probably have noticed, the Google UI is going through some changes as we try to add increasing amounts of capability without making the whole thing bog down with a million additional widgety items on the search page.  


It's a design challenge, that's for sure.  What we're trying to do is to improve the searchability of the web--usually that means adding new features for search (such as time-based restrictions) or location-based search (which you can do by using the "Change Location" widget in the left-hand panel).


Recently, the "Advanced Search" button moved from where-it-always-was (right by the search box) to under the gear menu in the upper-right corner of the search page.  



When you click on the gear, it will show a pull-down menu that looks like this (although with your name and your picture instead of mine).  



Note that this is ALSO where the Language tools ended up.  That's where you can Search in another language,  Translate text, Translate web pages, Change your interface language.   


I've read a lot of people who aren't able to find the Advanced Search button (and now you know!), and a few others who are complaining because it's now too complicated. 


Yes, the location moved, and you have to do one additional click to get to the page.  But really, this isn't much of a barrier.  It's different than what you grew up with, so it seems a little odd.  But really, it's not much of a hassle, unless it changed while you weren't looking. 


What you should know about web applications:   Because web apps (like Google search) run in a browser, it's really easy for the development team to change them while you're not looking.  Next time you re-load the page, BAM!, a new UI can be there.  


Mostly that's for the better: they can fix problems before you even notice them and they can evolve an application in ways that just weren't possible in the days when apps came on CDs and suffered a massive change every 18 months or so.  For instance, when Search by Image came online, it was with a tiny UI change (the camera icon now appears in the Image Search box)--but it lets you do something very, very powerful.  


The bottom line for us users is that change is now the new norm.  Stuff will change in all web-apps (except for those that have stopped all development) on an ongoing basis. 


One of the skills of a literate user of the web is the ability to understand how things change and be able to both carry their previous skills forward into the new version, and to understand what new possibilities there are now.  


We searchers need to lead the charge in teaching everyone how the world changes.  


Search (and teach) on! 



Friday, November 4, 2011

Answer: How much did you understand when you first read Moby Dick?


The question was about the whaling ship in Moby Dick:  


     Captain Ahab's ship had a very strange design
     that made the upper deck seem like a whale's mouth. 
     What was it that made this striking appearance?  




This question came up because I was talking with a friend about the story of Moby Dick, and the conversation took a turn towards the symbols in the book.  We remembered there was something odd about the ship's design, but couldn't remember what.  


Try to find the answer to THAT original question  ("There's something odd about the Pequod, what is it?").  That is, so to speak, a tougher kettle of fish to answer.  But I couldn't ask you to solve that problem because our question was inarticulately wrapped up in what we thought of as odd.  "Odd" in something about the ship.  It's a case of "I'll know it when I see it..."  


So when I wrote this challenge question, I included the (big) hint of "upper deck" and "like a whale's mouth" in my description.  I didn't have that when I started.  YOU were lucky.  


But here's what I did.  


I realized quickly that searching for things like [ odd ship design ] and [ unusual design Pequod ] wasn't going to work.  I was hoping that someone might have written a senior thesis on the topic and those phrases would have hit one of those.  But no dice.  


A search was going to have to involve the actual text of the book.  I then realized that I could just download the entire text of Moby Dick and do a manual search through the text of the document, looking for places where the author wrote about the ship. 


Where can you get the full text of Moby Dick?  Don't bother looking in Google Books, they don't let you download the full text.  You've got to go to a resource like Project Gutenberg which holds full texts of many out-of-copyright books, including many classics like Moby Dick.  A quick search for [ Project Gutenberg ] (or [ download text Moby Dick ] ) and I had a copy on my hard drive.  At 1.3Mb, it's a small download.   


I got the HTML version of the book so I could do my searching in Chrome.  


A little unsure where to begin, I started looking (using the infamous Control-F method) for places in the text where Melville was describing the ship.  


Now the great thing about Control-F in Chrome is that the scroll bar shows you all of the hits throughout the entire document as little tick marks in the scrollbar.  Since in this case the document is all of Moby Dick, this shows the occurrence of my first find (for the word "ship") and the locations of all other instances of the word  throughout the entire novel.   If you look closely at the find box (top) you'll see there are 717 instances of "ship."  A few too many to check out.  
  
It's pretty clear that this word wasn't going to work--there are 717 hits for the word "deck" in the novel.  The question is what would work to find me this "odd ship design" passage?  


Luckily, it's pretty quick to test words that are probably involved in a passage about the design of the ship.  I tried words like "deck" and "design" without much luck.  (Too many hits.)  But then I tried the word "features" (in the sense of "design features")...  To be honest, I didn't really hold out much hope for this, but it worked!  There are only 11 occurrences of "feature" in the entire document, and the first instance of the word was in the following paragraph (Chapter 16, "The Ship"):   (highlighting mine)


You may have seen many a quaint craft in your day, for aught I know;—square-toed luggers; mountainous Japanese junks; butter-box galliots, and what not; but take my word for it, you never saw such a rare old craft as this same rare old Pequod. She was a ship of the old school, rather small if anything; with an old-fashioned claw-footed look about her. Long seasoned and weather-stained in the typhoons and calms of all four oceans, her old hull's complexion was darkened like a French grenadier's, who has alike fought in Egypt and Siberia. Her venerable bows looked bearded. Her masts—cut somewhere on the coast of Japan, where her original ones were lost overboard in a gale—her masts stood stiffly up like the spines of the three old kings of Cologne. Her ancient decks were worn and wrinkled, like the pilgrim-worshipped flag-stone in Canterbury Cathedral where Becket bled. But to all these her old antiquities, were added new and marvellous features, pertaining to the wild business that for more than half a century she had followed. Old Captain Peleg, many years her chief-mate, before he commanded another vessel of his own, and now a retired seaman, and one of the principal owners of the Pequod,—this old Peleg, during the term of his chief-mateship, had built upon her original grotesqueness, and inlaid it, all over, with a quaintness both of material and device, unmatched by anything except it be Thorkill-Hake's carved buckler or bedstead. She was apparelled like any barbaric Ethiopian emperor, his neck heavy with pendants of polished ivory. She was a thing of trophies. A cannibal of a craft, tricking herself forth in the chased bones of her enemies. All round, her unpanelled, open bulwarks were garnished like one continuous jaw, with the long sharp teeth of the sperm whale, inserted there for pins, to fasten her old hempen thews and tendons to. Those thews ran not through base blocks of land wood, but deftly travelled over sheaves of sea-ivory. Scorning a turnstile wheel at her reverend helm, she sported there a tiller; and that tiller was in one mass, curiously carved from the long narrow lower jaw of her hereditary foe. The helmsman who steered by that tiller in a tempest, felt like the Tartar, when he holds back his fiery steed by clutching its jaw. A noble craft, but somehow a most melancholy! All noble things are touched with that.


There's the answer:  the Pequod was decorated with the teeth of sperm whales all around the top of bulwark on the topdeck.  (And the tiller was crafted from a single piece of whale jawbone...)

So.. what's the take-away message from a SearchResearch perspective?

First, that sometimes you really have to dig into the material at hand.  Since I really didn't know quite what the correct section was, I really didn't have anything much to go on.  So I started a fishing expedition with terms that I thought would be likely to be involved.

Key to this iterative search, though, was using a fast finding tool that would both find the search term (whatever word I was looking for) AND give me an overview of the entire document with all of the hits.  If I'd had to sequentially check all of the 717 hits for the word "ship" one-at-a-time, I would have given up longer before finding the answer.

You have to have sharp tools to investigate deeply.

Search on!

Wednesday, November 2, 2011

Wednesday Search Challenge (Nov 2, 2011): How much did you understand when you first read Moby Dick?

If you're like me, you probably read Moby Dick at some point in your life.  Maybe in high school, maybe for general reading later on in life.  Regardless of when you read it, you probably have a pretty good mental image of the story, right?  Captain Ahab and the search for the great white whale... maybe something about the inherent qualities of mankind, some ideas about man vs. nature, madness and a bunch of characters, the likes of which would never enter a Starbucks, but would prefer their grog on the open sea.  


Whenever it was that you read Moby Dick, how much of it did you understand?  Let's test your level of understanding:  You know what a right-whale is, you probably remember the difference between a top-sail and a stun-sail, and you remember the characters of Starbuck, Queequeg and Ishmael, right? 






Bu what about the ship itself?  Most of the book takes place aboard ship, and in some ways it's as memorable a character as the people.  


So let's see if you can answer this question:  


     Captain Ahab's ship had a very strange design
     that made the upper deck seem like a whale's mouth. 
     What was it that made this striking appearance?  




Tomorrow I'll talk about how I figured this out, and how you might do the same. 




By the way... If you're enjoying these weekly search challenges, you might ALSO enjoy AGoogleADay.com where I run a daily version of these search problems.  



Search on!