Friday, January 24, 2020

Dataset Search is out of Beta!


At long last... 

... Google's Dataset search tool is now out of Beta and ready for your consideration.  

You can use it to find open-access datasets.  Just search on the topic of interest and you'll see a list of results on the left-side, and a summary of the dataset that's selected.  For example, here I'm searching for datasets about snakes... 


You can obviously scroll down the list on the left and see other datasets on this topic.  

If you click through to the dataset you'll see the original public, with links to their data. 

A couple of things to notice: 

1. Filters

On the upper left of the page, you'll see a few filtering options (Updated date; Download format; Usage rights; Free).  If you need a particular kind of data, use those to limit your results to what you need. 



2. Autocomplete options 

As you're entering your query, don't miss the popup autocompletions that will appear.  (If you're looking at your hands, you'll totally miss this... yet another reason to learn touch-typing!)  

Here's an example:  



This is a great way to see what other dataset searchers are looking for (as a community).  Perhaps it will give you some good ideas for your own search.  

As it stands now, Dataset search indexes around 25 million datasets.  Surely there's something of interest to you in there!  

Here are a few examples I found that I like (you can imagine interesting uses for these):  

a.  Annotated database of cat images with facial features tagged 
b.  Dataset of MIDI tracks of musical performances (MAESTRO)
c.  Crime and incarceration in the United States 
d.  Parrotfish density data.  (You knew I'd search for this, yes?)  


Explore, and enjoy! 

Search on.  

Wednesday, January 22, 2020

SearchResearch Challenge (1/22/20): Where is this place? Where do the people live?


Every so often a mystery appears... 

... and if you're a SearchResearcher, you have to ask What's going on here?  

Of course, that led us to a tough Challenge last time.  So THIS time, let's follow up with a somewhat simpler set of Challenges.  

Here's what happened to prompt this week's Challenge...  

As I was flying back to North America from southern Europe, I happened to look out the window when we were in the middle of the Atlantic. 

Down below I saw a few islands glinting greenly on the  sea.  Really?  What could be down there?  It was kind of a surprise because it's often cloudy--but I spotted a few islands in the blue.  

Here's where I was at--in the exact center of this image (not a photo, but a satellite composite from Google Earth).  I could see islands below, but didn't know what they were.  Unfortunately, the in-flight tracking system available to the passengers wasn't working on that flight. 




I figured it out later, but I thought this would make a nice Challenge for today.  

1.  From this blue satellite image, can you figure out what island archipelago I was flying over?  

2.  After I found out what the islands are, I did a bit of reading and found that a favorite dish is cozido, which has an unusual method of cooking.  What is that cooking method?  Are there other countries that also cook with this unusual method?  

3. This island group is pretty small, but the people are famous around the world.  It might seem an obvious question, but where do most people from these islands live?  

Have fun with these.  It shouldn't take long, and I found the answers pretty surprising.  

Be sure to let us know how you found out.  

Search on!  

Tuesday, January 21, 2020

Answer: Mysterious rainbows and fossils in the floor?

I should have known....  


.... that this SRS Challenge was more complicated than I'd expected.   I've been working on it for the past week!  


As I wrote 2 weeks ago... 


Rainbows are fairly straightforward.  Here's a nice one of my building (the brick and glass building where the rainbow ends).. 




But other rainbows get more complicated.  Just to remind you, here's the rainbow pattern that we were asking about last week.  



Floor rainbow 1.

So I'll put these questions to you, my faithful SearchResearchers...

1.  What's causing the rainbow effect on the floor in the photo above (#2)?  Any ideas?

Now I have to admit that this is a fiendishly difficult Challenge.  My apologies.  I didn't really give you enough information to figure this one out JUST by doing online searches. And that in itself is a useful Search Lesson. (More on that in a bit.)  

That rainbow is clearly NOT in the window, but is a property of the reflection on the floor.  Let's zoom in a bit.  Here's the rainbow pattern up close: 



As you can see, it's actually a repeated pattern, from the left: a standard red/yellow/green/blue rainbow, then  a red-orange section, then a dark band between the leftmost and the rightmost rainbows.  


I went around the office taking many pictures of the floor rainbows and everything was fine until I took this photo (one floor above "Floor rainbow 1").  



WHAT??  

It's the same kind of flooring... why would it not have a rainbow on it?  


I puzzled about that for a while, then the next morning I got this photo from a different part of the building: 




This floor is also a kind of fossiliferous stone, darker in color, but very similar to the stone flooring in the other images.  

So why is there a rainbow pattern in the center section and not to the left or right? 


Look carefully at the picture:  the center section (where the rainbow is) is actually a reflection of a window and stone paneling on the outside of the building.  Both the left and right sections are actually just reflections of the sky.  


Fascinating.  Why would that make a difference?  


One last bit of data:  I also noticed that the strength of the rainbows depended on the time of day.  


I ended up asking a friend of mine Jim who happens to have  PhD in optics.  (Thanks, Jim!)  


He wrote that:  


The light from the sky is polarized (at least partly) due to scattering. While light reflected by the floor at an angle acts also as a polarizer.  (Think of the glare from the windshield of your car being mostly one polarization and your sun glasses blocking that same polarization.)

When you put a piece of glass between these crossed polarizers most of the light should be blocked... except the window has stress in it rotates the polarization.  So the dark band is from a section of glass with no stress on it.  But the rotation of the polarization is dependent on wavelength. 
So on both sides of the dark band are white bands since most of the colors having their polarization rotated to pass the second polarizer (known as the analyzer) but by the time the birefringence becomes enough to rotate enough to get another dark band, the colors start to get out of synch.  As you move further away from the dark band,  you miss the blue then the red is missing then you get pink and other colors that are not colors of the rainbow but rather mixtures of colors!

You might try pushing on the window to see if you can modify the stress pattern or add some Scotch tape and see the effect.

Did I do that?  Of course!  Guess what?  The pattern moves on the floor, and in particular, if you push hard on the window, MORE rainbows form.  Why?  

I'll spare you all the details, but I took a LOT of photos of places where I saw the rainbow pattern and places where I did NOT see the pattern.  It took a lot of reading (starting with what Jim wrote above) before I figured it out.  

Here are the diagrams I ended up drawing for myself to make sense of it all.  Bottom line:  The windows and floors are basically a polarimeter in the wild. 

A polarimeter has light going thru a polarizer (2) then passing through an object like a piece of glass (3) and then through another polarizer, also known as the analyzer (4).  When you look at the output (on the right), stresses in the glass or plastic (3) show up as rainbow patterns.  P/C to Imgbin.  


A polarimeter is an optical device using polarized light that scientists use to see the stress patterns in clear glass or plastic.  Here's what those stress patterns look like when seen with polarized light.  The person is holding a plastic fork between two pieces of special polarizing plastic.  See the colors?   Look familiar?  The picture below is basically a handheld polarimeter.  

P/C to the Exploratorium.  

So, why are we seeing rainbow patterns like this on the floor in my office?  

The answer is hidden in Jim's message to me.  There are two things to know:  

(1) sunlight from the sky is somewhat polarized... depending on direction and time of day.  (I did a bit of searching on this... a great query is [ polarization of the sky ] which leads to several great articles pointing out that the sunlight is mostly strongly polarized at 90 degrees away from the sun.)   

(2) light that reflects off something is polarized.  That is, the floor is acting as the second polarizing filter.  


In essence, this in a naturally occurring polarimeter.  The atmosphere is polarizing the sunlight before it hits the window.  Then, the light bounces off the highly polished floor, which acts just like the second polarizer (that is, the analyzer).  The guy on the right then sees the rainbow patterns on the floor which are actually the stress patterns in the window glass.

So why doesn't this work in the dark-stone floor picture?  A: Because in this case, the sunlight is coming straight in from the sun... and it's not polarized!  By contrast, the light in the center section is reflects off the windows outside, which polarizes the light before it passes through the window, and reflects off the polished floor!  

One last mystery:  Why don't we see the rainbow on the "no rainbow" floor?  A:  It's too dirty!  That floor hadn't been cleaned in a while... there's no wax on it and the reflectivity is too low.  (I'm trying to get them to clean it, just to test my hypothesis.)   
2.  What ARE those things in the floor?



So... what are they?  






This one was pretty straight-forward... the obvious query:


     [ spiral shaped fossils in flooring ] 

Leads to this image search result: 


And once you know that, it's easy to check this result and by searching for: 

     [ ammonite marble flooring ] 

and find lots of examples of ammonites in the floor.  (But in reading the descriptions of the tiles, it's clear, as several Regular Readers pointed out, that fossils don't occur in marble, but in limestone instead.  Marble is metamorphosized limestone that has undergone enough heat and pressure that the calcium carbonate crystals have re-crystallized, wiping out any fossils that were in the limestone.)  

So those horn-shaped fossils are ammonites, a kind of extinct marine mollusk in the subclass Ammonoidea of the class Cephalopoda. These ammonites are more closely related to living coleoids (e.g., octopuses, squid, and cuttlefish) than they are to currently living nautilus that they highly resemble.  





Search Lessons



1. Sometimes you just don't have enough information and need to ask for more.  Once you realize this--that you have to seek out additional information about the situation--you know you're onto a real problem. A couple of intrepid SRS readers did that (and once they started asking more questions, that's when I realized this was harder than I thought)!  Real-life is often like this; you need to determine WHAT you need to know, and then set out to acquire that information.  So, although I didn't intend for this Challenge to be so difficult, it's a great lesson in the way real-world Challenges arrive on your doorstep... they might not have a neat, simple, elegant solution.  Sometimes they require more work, more data, more research.  

My apologies for giving you such a massively hard problem.  But kudos to those readers that worked hard to figure out an answer.  Collectively we got really close!  

(Plus, it pays to have smart friends you can ask questions!)  

2. Although the fossils in the floor was pretty easy, it pays to check and read the results carefully.  When I saw the floor, I really thought it was marble.  But reading the results taught me that fossils really don't occur in marble... but in limestone.  Double checking the results (and reading carefully) taught me all kinds of things!  




As always...    
Search on! 

Wednesday, January 8, 2020

SearchResearch Challenge (1/8/20): Mysterious rainbows and fossils in the floor?


We live in a remarkable world... 


.... if only you take the time to notice it.  Here's an example of what I mean.  

The other day as I was walking into work, it had been raining, and we had a glorious rainbow.  Here's my photo of that memorable walk.  The brick and glass building is my place in the Google-verse.  




(Yes, that's the green of the golf course that's next door to the Google offices.)  

As beautiful as that walk was, I was actually slightly more surprised when I saw another rainbow pattern... this time inside... on the marble floor.  That rainbow looked like this: 




I haven't done anything to this photo (no Shopping or anything)--that's really the way it looked as I was walking down the corridor.   (Note that this photo was taken a couple of hours after the previous rainbow-in-the-sky pic. It had stopped raining by then.)

If it helps, this is the view to the west--looking in the opposite direction of the previous rainbow.  And there's no oil or anything on the floor--it's clean and dry.  As you can see, there are FOUR different rainbow patterns here--and I found that to be fairly amazing.  Why?  

Interestingly enough, that same section of floor also holds another couple of mysteries.  See below.  These pictures are taken from the section of the floor that has the rainbow pattern on it.  (I don't think there's any connection here--lots of the floor in this building has these patterns.)  





As I walked around, I could find these strange spiral/horn-shaped things in the marble flooring.  

Even more remarkably, I asked a few folks who also inhabit my building about these things, and nobody seems to have noticed!  Zounds!  There are mysteries literally underfoot!  

So I'll put these questions to you, my faithful SearchResearchers...

1.  What's causing the rainbow effect on the floor in the photo above (#2)?  Any ideas?

2.  What ARE those things in the floor?

If you know off the top of your head, that's fine (but say so... and then give us a link to a credible resource that supports what you're saying).  

But for everyone else, be sure to tell US the search path you followed to figure out the answers.  

(I'd love to understand what's going on in the floor beneath my feet!) 


Search on! 



---- 
PSA: My book, The Joy of Search, is now available at fine booksellers (and online)!  

Friday, January 3, 2020

Answer: Difficult to find?


I'm often surprised... 

... when a Search Challenge turns out to be unexpectedly difficult.  Alas, this happens more often than I'd like.  You'd expect that by now I'd have a good sense for what makes a SearchResearch question easy or hard.  

To be fair, I think my assessment of hard vs. easy is correct about 90% of the time, it's just that pesky 10% of Challenges that keeps me on my toes.  The last Challenge of 2019 was like that.  

Here it is again:  


1. Can you find the full text of this article? 


Hebb, D. O. (November, 1978) “On watching myself get old”  Psychology Today, 12(6):15-23

This article came up in a conversation with my friend Pete, who mentioned that there were some great ideas in this article, and that I should read it.  

This was--I thought--going to be a quick search.  I'd planned on finding a magazine archive site and then just opening that particular volume from 1978... I'd expected it to be just a couple of minutes.  

Here's what happened instead: 

First thought: Maybe it's in the Google Books collection of Magazines.  They have a good collection of magazines, but it's a bit idiosyncratic--if they have what you seek, all is good.  For instance, Google Books has the complete run of Life magazine, New York magazine, The Advocate, Ski, Field & Stream, and so on. 

Here is a reasonably complete list of the magazine holdings in Google Books (about 140 items long).  

 But there's no Psychology Today!  

Where to next?  

I thought I'd check to see if the publisher / magazine has an archive.  The query:  [ "psychology today" archive ] quickly leads you to the Psychology Today archive... which, turns out to go back to January, 1992.  Not enough for our purposes.  

On the other hand, the site does offer to sell you back issues, but starting only with June 2003... there's nothing before that.  

But it did make me think that I should search for a place that sells magazine back issues.  

     [ magazine back issues ] 

takes you to a few different sites, but most notably the website BackIssues.com has a pretty deep collection--including our sought-after issue (volume 12 from 1978)... for $11.11 + shipping, which would bring it to $15.  (It would also take a week or so to get to me.)  

Obviously, I want to find this online and for free.  Is there a way? 

That previous query also suggested that I check the academic research service Questia.com for the magazine.  (Questia  claims to have more than 94,000 academic books and more than 14 million journal, magazine, and newspaper articles in their collection.  It also costs real money.)  

But when I check their holdings of Psychology Today, I see it only goes back to 2012--and that doesn't work for this Challenge.  

However, it DOES give me the idea to check other academic library data services.  
Sidenote: I promise that I'll write up an entire post about academic databases (what they are, what they cover, etc.)... but that will be for a future time.  

I logged into my university library site and found that lots of academic databases cover Psychology Today, but their coverage is often very limited.  I checked some of the big, well-known databases, but few of them go back to the '70s.  

I DID find that Gale's Health and Wellness database has entries for those dates... but when you drill down, you'll find it's ONLY citations (and not complete ones at that).  That is, you can find the article, but you can only see what kind of article it is and when published!  Frustrating!  

This is getting tough.  Does this exist anywhere online?  Maybe not!  

Next good idea?  Look for hardcopies (or microfilm) copies.  How do you do that?  I'll check WorldCat.org to find places close to me that would have the magazines from 1978.  

Here's WorldCat's interface: 



Clearly we'll want to use the "Advanced Search" option.  In that interface I specify the title, the years, and the format ("Journal/Magazine"): 



Once you do this, you'll get back a list of libraries near you (here, you can see my Zip code--but note that this works internationally as well and can point you to libraries nearly everywhere):



Here you can see that there are 5 libraries that have copies of Psychology Today, all within 3 miles (5km) of me.  

I know I can't get into the Sofia University library, so instead I hop on my bike and zip over to the Stanford Hospital Health Library (which IS open to the public).  

That's when I find out that I did a dumb thing:  I should have checked their holdings before going over!  ("Holdings" means what they actually have, either via online access or physically in the library.)  

If I HAD done that (by visiting their website and checking), I would have seen this: 



Which is fine:  They have access to Psychology Today in an eJournal format through Gale Health and Wellness... which we already know doesn't offer the full-text of articles before 1992!  Moral: Check the holdings before bicycling to the library. 

On the other hand, the librarian there mentioned that the Green Library (the main library on Stanford campus) probably did have it in hardcopy.  

Thankfully, although it's a private university, Stanford lets anyone search their online catalog, which is often a great way to find information about what you seek.  (I learned my lesson and checked before biking over.)  

In this case, I went to Stanford's online catalog and did the obvious search for their holdings on Psychology Today.  



As you can see, they have online access (again, nothing before 1992), but when you click on the "Check availability" button (gray button at the bottom), you get a new bit of information: 


Click to zoom in.

Hurrah!  They have hardcopies of the magazine, and they even have all of the issues going right back to 1978 and beyond.  

Even better:  See the blue button that says "Request on site access"?  That lets you request that the volume with that issue in it be brought out of long-term storage and be delivered to the library.  You can't take it out of the library, but you're able to scan it yourself. 

I'm not a Stanford student, and although I've taught computer science there in the past, I don't have any special privileges now.  But the Stanford library allows each person in the community to visit the library up to 3 times each year.  (Kudos to them for this!)

This method works to get the scanned version of the article, although it takes a bit of time.  I got it, scanned it, and life is good. 

But noticing that university libraries have collections like this leads to.... 

Another method to get the article!!  

Obviously, university libraries often have great archives of older non-scanned / non-full-text magazines and journals.  

As I've said before (and will doubtlessly say again), Keep any university library affiliation that you ever have... 

Why?  This is a perfect example.  

While I was waiting for the Stanford library delivery, I emailed to a few of my friends in other academic universities.  They went through the same process at their libraries (U. Maryland, UC San Diego, and U. North Carolina Chapel Hill)--each requesting a scan from the archives.  

To their credit, all three of my friends were able to come up with the scan in 24 hours (often by having a library worker pull out the appropriate microfiche and then convert that image to PDF--that's a service many university libraries provide). 

Bottom line:  I wasn't able to find a scan of the article anywhere on the web via regular search, I WAS able to get a scanned version through university archives (and a little human intervention).  All's fair in love and search!   

Here's the first page of the article.  I have the rest.  



Many thanks to my university library friends:  Nevenka (Maryland), Don (UCSD), and Gary (UNC Chapel Hill). It really pays to have a great, skilled, very literate social network that can help your SRSing.  



2.  Can you find the novel by James Patterson that has a yellow cover AND has a number in the title? 

This turns out to be fairly straight forward, although complicated a bit.  


I went to Images.Google.com to do a regular image search and then used the Color Filter tool to limit the covers to yellow... 


The result was a bit of a surprise: 


I didn't know that James Patterson has a LOT of books with numbers in the title!  

Doing a bit of checking on Google Books (or Amazon) quickly tells us that queries like: 

     [ James Patterson 1 ] 
     [ James Patterson 2 ] 

(etc.) 

... all yield books by Patterson up to 12 (probably beyond, but I stopped checking there).  

It's probably the case that MANY books by James Patterson have numbers, and many are yellow.  

Is there a list with all of Patterson's book titles? 

     [ list of books by James Patterson ] 

leads us to this list of books (from Patterson's website):  

1st to Die, 2nd Chance, 3rd Degree, 4th Of July, The 5th Horseman, The 6th Target, 7th Heaven, The 8th Confession, The 9th Judgment, 10th Anniversary, 11th Hour, 12th of Never, The 13-Minute Murder, 14th Deadly Sin, 15th Affair, 16th Seduction, 17th Suspect, 18th Abduction, 19th Christmas, 96 Words for Love

Well!  That certainly limits the range of possibilities.  Time to go back to my friend Bill and ask for a BIT more information.  There are 19 books here... surely he must have a bit more information to help out on our search!  


Search Lessons 


There are a bunch to take away here.  

1.  Not everything is online!  I know it's a surprise to some of you, but there are useful articles that were written before the age of the digital, and much of them haven't yet been digitized.  (The back catalog is really huge.  Maybe we'll get it all scanned in the future, but for the time being, you have to remember this.  It's NOT all online.)  

2.  One investigation leads to another.  In the long story of finding the article, notice how trying one thing often led to another idea about something to try.  Here's the recap of what I did:  
Google Magazines led to...
Psychology Today magazine archive led to...
searching for back issues led to...
Backissues.com led to...
Questia.com led to...
other academic sites led to...
Gale’s Health and Wellness led to...
WorldCat led to...
Stanford Health library led to...
Stanford Green library led to...
other academic libraries led to..
emailing librarian friends... 
That's a lot of steps, but notice that most of the steps were pretty quick.  The takeaway is that one idea often leads to (or is suggested by) something you're doing now.  In the case of my search on Backissues.com, it was an advertisement that reminded me of Questia and all of the other online databases.  

3.  Check the library listings carefully BEFORE you travel.  Don't do what I did and just travel to the library--the online catalog can save you multiple trips IF you use it before you travel. Make sure they really have what you want.  

4.  You have friends in libraries who can help out.  In this case, I have wonderful friends who can help with access to otherwise difficult to find content.  Make friends with your librarians--they are often your best internet search resource!  

5.  Remember that the search idiom [list of ... ] is a handy thing.  People love to make lists--they'll make lists of the craziest things. (My favorite list-of is the Wikipedia List-of-Lists-of-Lists.  It's always entertaining reading.)  


Hope you found this search saga interesting.  I certainly had a good time.  

Search on!