Answer: Finding things with additional property limits (beginning web scraping)

In this week's Challenge ... 

I asked two questions that seem different, but are really both examples of web scraping.  When I realized they could both be done using the same tool... well, that's too much of a chance to demonstration how to use web scraping in your day-to-day SearchResearch! 

I asked these questions which are both "can you grab this data from a web site, and then make these charts?"

1.  Can you find (or create) a table of 50 summer internship positions in cities that are in Silicon Valley, and not in San Francisco?  Ideally, you'd make an interactive map (like the one above), where you can click on the red button and read about the internship.  

2.  Can you make a chart like this one showing the price distribution of all the sofas in the Ikea catalog?  (With the current catalog data.)  
 

In the comments, Ramón got us off to a good start by pointing out that both of these problems required grabbing data from websites, which he pointed out is called "web scraping."  That's how I started this Challenge solution as well, by searching for web scraping tools.  (I'll write another post about web scraping generally... but this is what I did for this Challenge.)

     [ web scraping tool ] 

As you can see, there are many, many tools to do this--I just chose the first one, Import.io, as a handy way to scrape the internship data and the Ikea catalog.

Web scraping is just having a program extract data from a web page.  In the case of Import.io, you just hand it a URL and it pretty much just hands back a CSV file with the data in it.


So, for the first Challenge (finding cities with internships about "big data" that are in the Bay Area, but not in SF), we first have to find a web site that has internships listed.  My query to find them was:

     [ summer internships Mountain View ] 

Why Mountain View in the query?  Because it's pretty centrally located (and it's where I go to work everyday, so I have a local's interest), and I know there are a lot of large companies there with internship possibilities.

Looking at the list of results, I found Indeed.com, Jobs.com,  InternMatch.com (and many others).

I decided to scrape the LinkedIn.com listings--again, because they're just down the street, and I happen to known they have a large interest in "big data" topics.

Their jobs site is:  www.linkedin.com/job/  -- I just filled out the form for internships near Mountain View on the topic of "big data" and found a nice set of results.

Note that I set the "Location" filter to be "Mountain View" (I could have set it to any city in the Valley).  

To scrape this data, I just grab the URL from the page:

https://www.linkedin.com/job/intern-%22big-data%22-jobs-mountain-view-ca/?sort=relevance&page_num=2&trk=jserp_pagination_2

And drop that into Import.IO -- and let it scrape out the data.

At the bottom of the Import.IO page there's a "Download..." option.  I saved this data to a CSV file, and then imported that into a Google Spreadsheet, which gave me: 

As you probably noticed, this is only 25 positions.  I can easily go to the "next page" of results on the LinkedIn site, copy the URL, drop that into Import.IO and repeat.  I did this, and appended the results to the bottom of the spreadsheet.  (Shared spreadsheet link.) 

In the spreadsheet you can see that I copied all of the city names and put them into Sheet 3 ("Cities").  I did a quick cleanup there, and created a second column "CityName" so I could then pull this spreadsheet into a new My Map.  But FIRST I copied all of the Cities and CityNames into another spreadsheet  with JUST the names of the cities in it.  (Why?  Because My Maps doesn't like to import spreadsheets with special characters in the columns.  A fast workaround is to just make a new sheet with just the data--the city names--that I care about.)  

So.. my new (city names only) spreadsheet looks like this: 

And when you import that into a My Map, you get this map: 

Note that there are many more cities in the spreadsheet than are shown here--there are a lot of duplicates.  (I guess we could have looked-up the street addresses for each business, but that's too much like real work.)  



And now that you know this method....  

doing the Ikea sofas is just as straight-forward.  

Go to the Ikea page and search for sofas.  Grab the URL and paste into Import.IO -- that gives you another data table.  Their URL looks like this:  

     http://www.ikea.com/us/en/search/?query=sofa&pageNumber=1

Notice that this is page 1 out of 36.  It's kind of a hassle to get all 36 (but I'll write up how to do that in another post!), so let's get a few more and do the same "save as CSV."  

Luckily, Import.IO has a "Save 5 pages" which automatically grabs the next 5 pages of Ikea data (just by changing the &pageNumber=1 argument in the URL above.  

So by the time you save the CSV and import it into your spreadsheet, you'll have 120 rows of data.  (For completeness, you could go to page 6, and then import the next 5 pages... but I'm happy with 120 samples of Ikea line of sofas.)  Here's my spreadsheet:  

And then you can easily copy out column L (prodprice_price) and do whatever kind of visualization you'd like, including the chart like that above, the chart below, or others: 

Such as a histogram of prices (in $150 / bucket price-ranges).  You can see here where the bulk of their product lies.  Ikea has a target audience in mind, but also carries a few rather expensive items as well.  

Search Lessons:  

1.  Find the right tool.  As I've said many times before, often the best way to start a complex project is to figure out what operation you're actually doing.  (This particular task is called "web scraping.")  And then find a tool that will help you out.  In this case, Import.IO is perfect for the task.  Be aware that there are many such tools--some of which might be better matches for the task you're doing.  (It all depends on the details of what you're trying to do.) 

2.  Linking tools together.  To solve this Challenge, you needed to not just extract the data, but also load it into your favorite spreadsheet, do a bit of cleaning, and then either visualize it with your spreadsheet charting tools, or export/import your data to My Maps (or whatever your intended end goal is).  


Hope you enjoyed this Challenge as much as I did! 


Search on! 




______ 
Addendum:  In response to a couple of questions from readers, I went back and made the map of "big data internships" truly interactive.  Now if you click on a pin, you'll see the city, the company with the position, and a link to the job posting.  

Search Challenge (2/25/15): Finding things with additional property limits

THIS WEEK.... 
I had two questions that came up that superficially look very different, but upon reflection, I realized it's the same search challenge in both cases.  

Earlier this week the son of a friend asked if I could help them find a summer internship that would involve working on the topic of Big Data "somewhere in Silicon Valley, but not in San Francisco."  He went on to ask if could be within an easy commute of Redwood City (since that's where he's going to live this summer).  

I thought about it for a while, and was able to fairly quickly make a map that looks like this: 

where each red pin shows a possible summer internship position working on "big data."  (Interestingly enough, this is pretty much the map of the cities of Silicon Valley...)  

1.  Can you find (or create) a table of 50 summer internship positions in cities that are in Silicon Valley, and not in San Francisco?  Ideally, you'd make an interactive map (like the one above), where you can click on the red button and read about the internship.  

And then..... the very next day, a different friend said she was frustrated looking at  the Ikea catalog.  She's trying to buy a sofa, and found the range of options pretty overwhelming.  It's a great asset to have many things to choose among, but it's sometimes kind of a lot. 

She wondered to me,  "I just want to know the range of prices of Ikea sofas!"  In talking with her, it became clear that she was also really interested in what the distribution of prices is.  (That is, she wanted to know if all Ikea sofas are expensive, or if they have just as many economy-priced sofas as well.)  

I fairly quickly whipped up a chart like this one (not the actual chart, but it looks a lot like this): 

Here the X axis is just different model numbers, and the Y axis is the price.  So you can immediately see that about 25% of all their models fall in the $200 - $400 price range, with a bit more than half being priced below $200.  Obviously, the chart for sofas will be different (everything is probably more expensive).  

2.  Can you make a chart like this one showing the price distribution of all the sofas in the Ikea catalog?  (With the current catalog.)  

Finding the prices isn't hard.  (Ikea.com)  The question is how do you extract the prices (or internship position descriptions) and then do something with THAT data?  

Big Tips:  I know this seems like a crazy hard problem, but it's really not. You just have to know the right tools.   You should NOT spend much time (if any) copying and pasting data from the online catalogs of jobs or sofas.  You should be able to find a tool to help you do the automatic extraction of data from a web page.  

(If nobody's figured out how to do this by the EOD tomorrow, I'll give you another big hint on Thursday.)  

A bit o' philosophy:  This is yet-another of Dan's "find the data and massage it" Search Challenges.  As I've said before, this is a blog about Search and Sensemaking.  Although "sensemaking" is typically a larger, longer behavior pattern, these "find the data / massage it" kinds of questions are typical of the kinds of sensemaking questions that professional analysts have to solve all the time.  Because we're trying to have fun AND learn something, my Challenges don't go on for weeks or months, but try to give you the sense of what the larger skill set is like.  

So I hope you enjoy these "find & massage" search data challenges as much as I do.  In truth, I'm having a good time creating these Challenges that teach a very particular skill, and sometimes give a bit of insight at the same time.  

Search on! 

Answer: A couple of odd questions...

One week, two curious questions.   Two interesting answers... 

1.  Those equations are really interesting, but WHAT do they mean?  And why would Woody Paul put them on his concert performance gear?  (For extra points: Where did Woody go to college?)  

Woody Paul's awesome shirt with mysterious equations.

It's not hard to figure out from a quick query who Woody Paul really is: 

     [ Woody Paul "Riders in the Sky" ] 

Here I quoted the name of the band just to make sure I didn't get anything spurious.  (It turns out not to matter too much.)  

Woody Paul is his stage name.  Real name:  Paul Chrisman. My favorite article was from MIT's Technology Review magazine which points out that he got his PhD in nuclear engineering from MIT in 1976.  When he graduated there were two job offers waiting--an assistant professorship at Columbia University or a recording gig in California.  (Other versions of this story have him heading to Nashville.) It's clear he chose music in either case. 

A little more poking around finds that his thesis was "Inertial, Viscous, and Finite-Beta Effects in a Resistive, Time Dependent Tokamak Discharge", Thesis Nuc. Eng. 1976, PhD, supervised by James E. McCune. 

This is all relevant because it gives us a clue about what these symbols on his shirt might be. 

If you already recognize the symbols involved, you've got a headstart.  But suppose you DON'T recognize any of these symbols, how do you start?  

You might remember that earlier I've written about "Symbol Search" using either the Shape Catcher web app, or the Google Docs symbol search tool.  

In both cases, you draw the character and look through the list of recognized symbols.  Here's my screenimage from using ShapeCatcher.com  (it works with Google Docs symbol reco as well)... 

 The trick here is to notice (look carefully!) that the symbol on Woody's shirt has a special, thickened left side on the downward pointing triangle.  That's what makes it a "nabla" symbol.  (Note:  It's not a delta--that's a triangle that points up, not down!) 

Once you know that, you can do a query for: 

     [ nabla ] 

and learn that it is the name of the symbol in mathematics that (quoting Wikipedia) "... is used in mathematics to denote the del operator, a differential operator that indicates taking gradient, divergence, or curl..."  

That might look scary, but hold on a second--stay with me here.  The key idea here is that nabla isn't the big thing, that's just the character's name (it's a bit like saying "virgule" for the slash, or divide, character).  The important term to notice here is that it's the del operator.  

Okay, so what's del?  

If you [ define del ] you'll find it's an operator used in vector mathematics.  

So now I just searched for the most obvious natural language translation of this, which I put down as: 

     [ del dot b equals 0  ]

Sure enough, once you do that, you land in the land of physics discussions.  

It only takes a minute or so of looking around there (in PhysicsForums.com) to find out that his "del dot E" is one of Maxwell's equations.  A quick search for that takes us to the Wikipedia entry (or any of a thousand textbooks, all with exactly the same information): 

Adapted from Wikipedia entry on Maxwell's Equations

As you can see, Woody has Gauss's law in his collar, Gauss's law of magentism and  Faraday's law of induction embroidered into his western-style, fringed shirt.  And as Luis cleverly spotted in an image I hadn't seen before, assume Ampère's circuital law is on the back of the yoke.  (You can just see it riding up over his right shoulder here.)  

Except of image from SCVhistory.com

  

Maxwell's equations are four equations that form the foundation of electrodynamics, classical optics, and electric circuits. They describe how electric and magnetic fields are generated and altered by each other and by charges and currents. These equations are named after the Scottish physicist and mathematician James Clerk Maxwell, who published an early form of those equations between 1861 and 1862.

They are, of course, equations that Woody would have used extensively in his thesis writings.  A Tokamak reactor is one that uses a strong torus-shaped magnetic field to contain a plasma.  It was (and still is, in some circles) the best approach for possibly generating a viable fusion reactor. 




2.  Have crows become suddenly much more common in the Bay Area?  Is it just crows, or have ravens also turned into frequent guests?  

You know, sometimes things don't work out the way you'd expect.  I'd fully expected that the only good way to answer this would have been to look at data pulled from the annual Audubon bird counts.  (Such as this entry for the 2014 counts of crows.) 

But people other than me have noticed this crow / raven trend.  Consequently, the first searches you might do:  

     [ crow population statistics san francisco bay area ] 

end up pretty much answering the question.  The lesson for me is obvious--pre-test the questions!  

Several people pointed to one of the three articles about crow population expansion.  

SFGate (a local media service) wrote an article in 2012 about "Why ravens, crows are more common now in the Bay Area" quoting an ornithologist from Cornell, and giving crow counts from the Audubon census of 1991 (17 crows and 54 ravens in San Francisco; 60 crows and 23 ravens in Oakland), and 2011 (SF tallied 566 crows and 599 ravens;  while Oakland had 1,152 crows and 193 ravens).  

They also pointed out that crows were "relatively rare" back in 1927, so this really is a recent phenomenon.  

The San Jose Mercury News also had an article that same year, except giving similar numbers for crows growth in the penninsula (where I live) and the South Bay (where San José is).  Their article, "Counting crows: Number of black birds on the rise in Bay Area" also has a lovely chart showing the growth rate over time.  

Chart from SJ Mercury story on crows, 2012, by Aaron Kinney.  

As you can see, the growth rate the South Bay (San Jose) has been nothing short of spectacular.  

But just this past month, the Mercury wrote another article about crows.  "They're everywhere! Crows, ravens overrun Bay Area" (Nicholas Wieler, Santa Cruz Sentinel, 2/14/15)  which repeats the same basic data, but adds a new graph showing crows v. ravens.  

Graph from SJ Merc article.  

The findings here are clear enough:  Yes, both crows and ravens have been skyrocketing in populations--with crows more common in the South Bay, and ravens more common in San Francisco.  


Search Lessons: 

1.  Remember the search tools you know about!  Finding the nabla is easy, IF you know about the symbol finding tool.  (And if you don't, try describing it in the simplest way possible-- something like:  [ downward pointing triangle symbol ]  )  
2.  Once you've found the symbol name, trying searching for the way it's commonly used.  In this case, it was to discover that the symbol was also called "del" and under that name, it's easy to find in the physics literature. (Which is why it was helpful to know about Wood Paul's previous writings...)  

3.  When searching for analyses (e.g., crows population over time), always search for a completed report.  You never know (I certainly didn't!) when someone will have already done the analysis for you.  Double check the data and the sources, but all of these articles refer to data from Audubon Society and/or the Cornell Lab of Ornithology--both extremely respected resources in the birding world.  

Search on! 

Search Challenge (2/18/15): A couple of odd questions....

Every so often you see something and just stop to wonder about it.  This happened to me twice this past week, and in finding the answers, I realized they'd make good Challenges for you to consider.  

I stayed at home this past week with a truly annoying case of bronchitis.  I was functional--barely--but rather than go into work, I stayed at home the entire week, working via email, Google Hangouts, and shared Docs.  It's not a bad way to work, truthfully (although I did miss the Google cafeteria).  

But as I sat at home I noticed two things that piqued my curiosity.  Can you figure out the answers to these questions?  

While listening to an album by the cowboy group, Riders in the Sky, I picked up the CD case (from which I'd ripped the tracks) and spotted something that stood out.  As you know, the Riders in the Sky group play in the style of the classic cowboy and Western songs from the 1930s--it's a celebration of the mythical cowpoke songs that were popular back then (think of Gene Autry, Roy Rogers, Tex Ritter--the guys in white hats with guitars).  

Looking at the picture of "Woody Paul," I spotted something on his shirt that seemed a bit out-of-place.  As you know, singing cowboy shirts are colorful affairs, usually with western motifs and decorations.  But this was different.  Here's an excerpt from his picture: 

Woody Paul's awesome shirt.
Image excerpted from Riders in the Sky website.

See his collar and over the left side of his chest?  Those are some kind of mathematics equations.  Really?  On a cowboy shirt?  

1.  Those equations are really interesting, but WHAT do they mean?  And why would Woody Paul put them on his concert performance gear?  (For extra points: Where did Woody go to college?)  

While working from home, I had the chance to sit outside in the backyard, laptop in lap, enjoying the warm sunshine and drinking lemonade.  The backyard has a fair number of trees, and over the years I've enjoyed watching the birds come and go with the seasons.  

Some bird species never really leave, they just hang around all year.  Some kinds of hummingbirds (Anna's, for instance), mourning doves, and crows are resident--they're here spring, summer, fall, winter.  

But oddly enough, I seem to be seeing a LOT more crows this year than in years past.  Now that I think about it, I used to rarely see them when I first moved to the San Francisco Bay area in the 1980s, then they seemed to have become more and more common over the years.  If I were to guess, I'd say that the past decade seems to have been a huge expansion period for crows here.  Is this true?

2.  Have crows become suddenly much more common in the Bay Area?  Is it just crows, or have ravens also turned into frequent guests?  

I'm really interested here in HOW you find out the answers to these questions.  It's not immediately obvious how to search for Woody's shirt equation, nor is it immediately obvious how to find the data for crow population growth.  So please be sure to let us know how you did your search!  

My answer on Friday!  

Search on.  

Answer: How did traffic signs come to be?

As you recall, a while back, a reader asked the question:  

1.  When did turn signals on AUTOMOBILES come to be a thing?  Can you determine when the first turn signals were installed on cars?  Who invented them? 

And that led me to wonder in turn... 

2.  When did signals on the ROAD come to be a thing?  Can you figure out when (and where) the first traffic lights were installed? 




This is a fun research Challenge--partly because it's interesting to see ALL of the different stories that are out there about traffic lights, but also because there are some key Search Lessons we have to remember to do this one well.  

As Rosemary, Ramón, Remmij, Debbie, Anne,  and Jon point out in their comments, these things were invented quite a while ago--it's very likely that the names would have changed over time.  

Like Ramón, I started with the query:  

     [ turn signals history ] 

which took me to the "Second Chance Garage" site and their story about turn signals (along with many of you, apparently).   In there, we learn that in 1909, a British man named Percy Douglas-Hamilton took out a patent on a "Device for indicating the intended movements of vehicles."  A quick Google Patent Search (that's a link to the Advanced Patent Search page) on his name gives us US patent #912831.  (incidentally, Percy Seymour Douglas-Hamilton lived at 6 Queen's Gate Gardens, London, at the time of the patent.) 

As you can see from the illustrations in the patent, the signals were in the shapes of hands.  The upheld hand would light up to indicate stopping, and the hands that point left and right would be back-illuminated to indicate a turn in that direction.  In reading the patent, it's clear that there would have been at least 3 different switches for the driver to use (one for stopping, one for each direction).  The lights would not have blinked.  

Continuing our search for the history of turn signals, I poked around the search results for a while, finding all kinds of interesting tidbits.  Including this piece from the New York Times ("Who made that turn signal?" July 12, 2013) that covers a number of signal introductions, and the relatively long amount of time it took for them to be adopted.  To quote from that article:  "Actress Florence Lawrence...developed an early version of the turn signal. “I have invented an ‘auto-signaling arm,’ which, when placed on the back of the fender, can be raised or lowered by electrical push buttons...”  But it didn't catch on, nor did the other inventions.  

Since this was so long ago, it's very likely that the terminology changed between now and then, so I did a quick Wikipedia search for: 

     [ Wikipedia turn signals ] 

and found that there was an Italian system of "trafficators" that dates back to 1908.  

"They [trafficators] first appeared in the 1900s, when they were actuated either mechanically or pneumatically. In 1908, Alfredo Barrachini in Rome had added electric lights inside the arms, that turned on as they extended, but operation was still by a cable system. Electric operation came in 1918 when the Naillik Motor Signal Company of Boston added electric motor drive." 

So the consensus seems to be that the inventions date to 1908, becoming more-or-less standard in the 1930s, with Buick introducing factory-installed turn signals (as opposed to after-market kits, which had been around for a while).  


Let's turn our attention to traffic lights:  When did THEY become a common thing? 

The obvious query: 

     [ traffic light history ] 

leads us to the Wikipedia article on the History of Traffic Lights.  The story told there is long and interesting, but the critical point is that on December 10, 1868, a semaphore system was put in place at the junction of Great George and Bridge Street in London, near Parliament.  Downtown London had so much horse and wagon traffic that a British railroad signal engineer designed the first traffic semaphore to regulate traffic movement. The device was the first traffic “light”, as lenses in the semaphores where lighted by gas lanterns at night. 

Armed with this information, I did a few queries: 

     [ 1868 semaphore system Parliament ]   (and similar) 

and found lots of other evidence to support this origin story.  One of the more interesting documents is from the Transportation Association of Canada annual conference, which tells this same story (and is a pretty reputable source). 

These sources also repeat the story of this first gas-powered traffic signal blowing up, injuring the policeman-operator.  

But for traffic signals as we commonly think of them today, I did a query for: 

     [ traffic signal history ] 

which led to several sources for the origin of the four-way, three-color traffic lighting system.  The red/yellow/green colors were used in the railway system of the day, and in 1920 in Detroit Michigan policeman  William L. Potts lighting scheme. As a consequence, Detroit became the first to use the red, green, and yellow lights to control road traffic.   See:  The Origin of the Green, Yellow, and Red Color Scheme for Traffic Lights  and the Wikipedia article above.    Traffic Lights Invented by William L. Potts  And for years afterwards, there was a period of great experimentation, including signals that copied some of the iconography seen on car turn signals as well.  

Image from FHWA.DOT.GOV

Search Lessons:  As I said, this wasn't a hard search problem (but it was extraordinarily interesting--I spent way too much time reading everyone's comments and clicking on all of the links.. .thanks!)  

But there are three lessons I want to point out today: 

1.  Wikipedia is a great place to start your research, but don't end there.  You know that (in general) the quality of the articles on Wikipedia is quite good.  But you should ALWAYS second source what you discover--especially when it's difficult, or unbelievable.  Be sure to avoid duplicative text--that is, whenever you see a word-for-word copy from the Wikipedia article, you can't really count that duplicate as an independent source.  

2.  Patent search is a great place to look for the start of a technology, but not its adoption.  There are many, many patents for technologies that began at one time, but then took decades to finally be adopted.  But that doesn't stop you from searching the Patents and discovering who did what, and what companies were involved.  You can often discover key players by looking at the patents.  

3.  When doing historical research, be very aware of the terms people used to describe things.  Remember the "trafficator" from above?  As you read through older documents, keep a sharp eye open for specific terms and phrases.  "Semaphore" and "signaling systems" kept coming up in my searches.  

A footnote:  One things that did NOT work for me (although I spent about 2 hours trying) is Book search.  I'm not sure why, but I just was not able to find a decent history of cars / automobiles that included any kind of discussion of the evolution of turn signals.  I tried "directional indicator" (and variations of that phrase, since that's the "classic" term used), but it just didn't work out.  

Did anyone else have success finding something in a book on this topic?  

Still searching! 

Search Challenge (2/11/15): How did traffic signs come to be?

This week's Challenge is another fun one that came across my desk.  It's a short, fun, historical romp through the fields of yesteryear.  

A while back, a reader asked the question:  

1.  When did turn signals on AUTOMOBILES come to be a thing?  Can you determine when the first turn signals were installed on cars?  Who invented them? 

And that led me to wonder in turn... 

2.  When did signals on the ROAD come to be a thing?  Can you figure out when (and where) the first traffic lights were installed? 


This is a fun one.  The search insight here will be something about what sources you choose to draw upon.  

How will you find out? 

When you know the answer (or at least, AN answer), be sure to write up a comment about HOW you found the answer.  What sources did you consult, and why those particular sources?  

Search on!  

New series: How-to-find...

I thought I might try something new here.

My plan is to start doing weekly (or biweekly, depending on how energetic I am) series of posts about "How To Find....<something>" (HTF)

The idea is that <something> will vary from week to week.

This week, I'm trying out my HTF notes about finding "Do it Yourself" information.

Please let me know what you think of this. I'm especially interested if you have additional things I should add to this "HTF DIY Information," or if you think the whole idea is a good one.

Do you have any particular topic you'd like to see in the HTF series?

Thanks.

-- Dan

++++++++++++++++++++++++++++++++++

How to find: DIY information 

Introduction

A common thing for people search for is “how to” information.  Sometimes called “do it yourself” (DIY),this kind of how-to-do-something is an important part of how people share their craft with others.  In the past few years, plenty of  web sites have sprung up to teach people how to sew, repair broken appliances, darn socks, or do thoracic surgery.  Once this kind of information was the area of hobbyists and obscure, difficult-to-find speciality magazines.  But now, the DIY and Maker movements have broken boundaries with some sophisticated DIY information that’s easily findable and widely available.  (Think about examples like  “how to build your own surfboard,”  “set up your own Minecraft server,” or “how to do fire spinning.”)  

Sometimes, getting the DIY information rapidly is critical--the water is gushing out of my plumbing NOW and I need to stop it instantly.  Most of the time, getting the DIY information is leisurely--you can learn how to fly a drone or build a Minecraft server pretty much any time.  In the get-it-to-me-now case, you don’t want to spend a lot of time futzing around… and that’s why you read this article now.  In the leisurely case, you probably want to find pretty reliable “how to” information so you don’t crash your drone on its first flight, or spent lots of time building a broken server.  

What is DIY content?  

DIY--or “how to do it”--information tells you (or better yet, shows you) how to do some particularly skilled thing.  Usually DIY info is for topics where it’s really not obvious how to do it (for instance, how DO you cut glass to make stained glass artwork?), mysterious (how do you make a fishing net out of a long string?), or involves steps where doing it wrong is really dangerous or expensive.

Lots of DIY content these days is in video form, although printed manuals and how-to guides are sometimes easier to use.  

While there are MANY kinds of DIY information, we’re going to look at just the most common kinds:

1. How to do a particular skill?  (Think twirling a fire baton, riding a unicycle, replacing car brake pads, or how to strum a power chord on your electric guitar at max volume.)  
   
2. How to fix something that’s broken?  (Your blender / TV / computer is broken.   Your socks need repair. Your kitchen faucet needs replacing.  What now?)
   
3. How to make something from scratch?  (Learn to bake a cake, build an igloo, make the best paper airplane, or write a strong resume.)  
   
4. How do you use a tool or piece of software?  (You need to learn how to fix up old photos using a software photo editor.  You’d like to learn how to use an awl correctly, without sticking it into your hand.)  

DIY searching: What do you need to learn?  

Luckily, the internet is full of people who have created tutorials and written-up how-tos for even the most obscure topics.  (Need to know how to take care of a pet spider?  There are tutorials written for you. Really.)  

Consider what you already know.  If you’re looking up DIY information about creating a new Mardi Gras costume, think about how much you already know.  Are you a sewer?  Do you have a closet full of needles and thread, ribbons and bolts of fabric?  Are you already an expert in the field?  

When starting a DIY search, first consider what kind of information you need.  If you’re a beginner, you’re going to need an overview or quick introduction to the field, if only to learn the language and to assess whether or not this is a good thing to start doing.  (It could be that you’re taking on something way over your head or budget.  That’s the point of up-front research:  Find this kind of thing out before sinking lots of time and money into a project.  Learning how to bake bread is fairly straightforward; learning how to bake a beautifully decorated cake involves more time, money, and practice.)  Check out the results all the way to the end.  (Don’t be surprised by a suddenly large amount of time you need at the end of the recipe when your dinner party is TONIGHT.)  

Once you’ve started finding your research, think about building up a collection of articles, evaluating which one(s) you think are the best.  Are they in language you understand?  Is it clear what’s involved?

Pro tip:  Always search for at least two or three different how-to articles (or vidoes) before diving in.  It’s often the case that one article will illustrate the method in a way that doesn’t make sense until you read another take on the same topic.  

DIY  search process

Start broadly:  When I’m doing a DIY search in an area I don’t know much about, I start broadly, usually learning a lot about the field before I dive into the specifics.  For instance, I know very little about knitting.  So if I wanted to get into knitting as a spare-time activity, I’d first look up more general articles about knitting to get a sense for what’s involved.  Use queries such as:  

    [ knitting overview ]

    [ introduction to knitting ]

    [ beginning knitting ]

I’d look at the high end to see the things I’d like to aspire to do one day, and then go back and look at the entry-level, or beginner’s level materials.  Can I get there from here?  

Dive in:  If I already know what I’m doing (or if I’ve learned a lot already), I’ll start to dive into mechanics of searching for teaching material.  I start broadly, casting out a wide net, and look for specialty sites along the way.  Let’s take the example of guitar playing:  

    [ how to play guitar ]

    [ guitar instruction ]

    [ guitar lessons ]

And if you know what style of guitar playing you’d like to pursue, add that in as well:

    [ how to play flamenco guitar ]

    [ gypsy guitar instruction ]

    [ jazz guitar lessons ]

Methods

In this section we cover 4  different methods for searching out educational content.  Broadly speaking, these methods are all ways to translate what you know about an educational need into search-engine specific strategies.  

1.  Use specific terms that are use in your interest area.  For instance, a cable weave is a kind of knitting stitch, while a cable braid is a way to manage all of those pesky computer cables under the desk.  A “caliper” is part of a car’s brakes, but also a machinists measuring tool.  You can use specific terms like this to get very on-target search results.  (Caution:  Be sure you know what your speciality term means!  Don’t search for “penny whistle” if what you’re really looking for is “recorder.”  Use [ define  ] to double check.)  

     [ cable weave knitting pattern ]

    vs.

     [ woven cable headphone ]

2.  Check out different kinds of media.  Remember that there can be many different kinds of content.  Often we turn to videos to find out how to do something physical (e.g., fix plumbing or learning a dance move), but printed documents can also be very helpful, especially when they’re specifically for the thing you’re trying to repair. Sometimes an exploded parts diagram that you can refer to is exactly the right thing.  Also look for images for your topic.  Electronics repairs often require a schematic diagram to help you understand how things are put together.  

    [ repair manual PDF Cuisinart blender ]   (will find PDFs for a Cuisinart blender)

    [ furnace schematic ]   (Image search)  

And while it might seem odd, remember that Books can be a useful place to learn how-to do something.  Be sure to checkout Google Books.  (Books.Google.com)  

3.  Look for Q&A or Forum sites.  A Q&A (questions and answers) or Forum site can be a superb source of information.  These sites are usually run by enthusiasts in that particular field to answer questions that come up for people.  

    [ forum tile repair ]

    [ Q&A bicycle repair ]

4.  Search for online communities in your interest area.  Many social media networks (Facebook, Pintrest, G+, Tumblr, etc.) have communities of people with a shared interest.  It’s simple to look on a social network for things like:

    [ piano enthusiasts ]

    [ woodworking ]

    [ surfing ]  

and get quickly linked into those communities, usually full of people who are more than willing to answer your questions.  

5.  Search for DIY content for your specific device / widget / gadget.  People love to talk about their particular gadget.  So it’s realtively straightforward to look for how-to information that’s keyed to a particular kind of device.  Notice:  Be sure the article you’re reading and the device you have are the same model (or release).  Nothing is more frustrating than reading an entire how-to article and then figuring out that this was all for the previous version of the device… that you don’t own.  

    [ GoPro Silver how to time lapse ]

    [  Photoshop CC tutorial ]

Specific DIY Resources  

There are many videos on YouTube for your interest area.  Be sure to look not just for individual videos, but also for channels that are on your topic.

Stack Exchange Q&A sites on many topics.  http://stackexchange.com/sites

Often a manufacturer will have a website that’s dedicated to supporting their gear.  (A couple of examples: GoPro http://gopro.com/support/, Seagate http://www.seagate.com/support/ , etc.)  

Popular DIY sites:  

WikiHow

instructables

eHow

HowToGeek.com

HowTo.com

MOOCs:  There are a great many MOOCs that will teach you specific things (e.g., how to code, how to do data analysis, how to play jazz guitar).  Although they tend to be longer format, they’re often a great resource for life-long learning.  

Lynda.com -- a commercial online education resource (costs real money), but some of their technical content is superb.  

Chilton’s auto repair manuals (available in many libraries)  

Don’t forget about university courses and classes at your local library.