Answer: Where is everybody moving to?

Where is everybody moving to? 

(Sorry this took so long to answer.. but... The question is easy to ask, but more complicated to answer than our usual research topics!!  

Our Challenge was..   

1.  During the past year, what have been the large movements of people across the world?  (We're interested here in people who are moving from one place to another for economic, political, or safety reasons--not people going home temporarily for the New Year's or Thanksgiving Day celebrations.)  

A. Find data sources that tell us how many people moved from location X to location Y in 2015? 

B. Is there a good way to visualize that data?  (Giant tables of numbers aren't all that descriptive, there must be a better way. Can you find a good one?)  

C. How do you know that these numbers are right?  And to what extent can you believe them?  Why?  

My thoughts on this:  First, this is a HUGE problem.  (Note to self--don't make such gigantic Challenges in the future! One could spend an unbounded amount of time on this. I also note that I probably should have asked for data from 2014, since some organizations still haven't published their 2015 data yet!)  

To answer this in a reasonable amount of time, I'm going to focus on getting the Mexico/US migration data, and extend that to Syria refugees, and then wrap up with a couple of information visualizations.  

Mexico/US Migrations:  I live in California, one of the border states with Mexico.  I was born in Los Angeles, and live now in Palo Alto, so the Mexican heritage runs fairly deep.  As you know, migration across the border is a very contentious issue, which means that getting solid data might be a little tricky.  

As Regular Reader Remmij said beautifully:  "...events outpace data collection > factual accuracy is hard to come by > data is subject to manufacture/agenda manipulation > generalities outpace specifics >
perceived data outweighs ground truth in a media driven environment"  

But let's give it a shot.  Clearly, our strategy will have to be to triangulate the data from multiple sources.  

I'll start with a query of: 

     [ migration data Mexico US ] 

and limit the hits to the past year (using the Search Tool drop-down "Past Year").  By looking through the hits, it's clear that there are two main sources of immigration data--Pew Research and the Census.gov data on immigration.  

As is obvious, there's a lot of debate about which data to trust.  Some think tanks find much of the data in dispute (e.g., Austin Institute), and you can find extended debates about how the data is collected (is it accurate?  what time period does it cover?  what's the sampling method?).  

But I know that both Pew and the Census.gov people try to do a reasonable job.  So let's look at their data first. 

Pew Research: In their article about Mexican migration give the following data: 

Figure from Pew Research report on net migrations to/from Mexico.

This chart only covers 1995 to 2014.  (Notice also that the middle bars cover the years 2005 to 2010, while the bars on the right ALSO covers 2009 - 2014... in essence, they overlap by one year.)  Still, you can see a trend beginning of net migration FROM the US to Mexico.  You can read about their census methods in their Methodology article. 

 In summary, they get data from the Mexican government (National Survey of Demographic Dynamics (Encuesta Nacional de la Dinámica Demográfica—ENADID): 2014) and from the US Census (Current Population Survey (CPS): 2000-2014 March Annual Social and Economic Supplement (ASEC)).  They do a fairly sophisticated analysis to get good estimates of the true number.   

In a different article, Pew points out that the net migration (by year) between Mexico and the US is primarily in the direction of Mexico.  

The article "More Mexicans Leaving than Coming to the US"  they publish a telling chart (see below).  This chart is based on the same dataset as above (NSDD, ENADID, US Census, etc.).  

How does the Census.gov data compare? 

I did a search like this: 

     [ migration US Mexico site:census.gov ] 

and found the Census.gov site for International Programs (which has data for other countries, including migration rates in/out of the US).  

Their key chart for this period (and a forecast out to 2025) is this: 

Figure from Census.gov, International Program.

Although it's not an exact copy of the Pew data, the trends agree: the net migration pattern is from the US to Mexico, not the other way around.  

Exploring this idea of "finding a reliable source," I did a query for: 

     [ DHS Mexico US migration  data ] 

thinking that DHS (Department of Homeland Security) would have reliable data.  And, unsurprisingly, I found their 2013 Handbook of Immigration Statistics.  

This is really a compendium of all kinds of immigration data (including emigration for all kinds of reasons).  But the relevant data are: 

"Persons obtaining lawful residence by country of origin"  

2013, Mexico:  134,198

"Persons naturalized, by country of origin" 

2013, Mexico:  99,385 

There are many other data tables here as well, including "Non-migrant temporary workers" and others that I'll skip since they're temporary status.  

But the other factor is "Aliens apprehended by country of origin" 

2013, Mexico:  424,978 

If you keep looking through this resource, you'll get pretty darn close to the Pew estimate from above (of 870K immigrants during 2013).  (And, FWIW, the historical trends duplicate the Pew findings--they're net towards Mexico!)  


I thought of another data source that might be reliable:  The UN.  My query: 

     [ UN Mexico US migration  data ] 

led me to a large data-rich report on the total number of migrants from each country in the world.  There I found a data table that reported that (overall) 1.1M Mexicans had emigrated to the US  in 2015 (but didn't report on the number who left the US to go elsewhere).  That's a little higher than the Pew estimate, but "only" by 200,000.  

More importantly, this reference, International Migration Report, goes a long way towards answering our other question about large migrations across the world, as there's data for migration from every country.  In the summary page, they write: 

"Nearly two thirds of all international migrants live in Europe (76 million) or Asia (75 million). Northern America hosted the third largest number of international migrants (54 million), followed by Africa (21 million), Latin America and the Caribbean (9 million) and Oceania (8 million). 

In 2015, two thirds (67 per cent) of all international migrants were living in just twenty countries. The largest number of international migrants (47 million) resided in the United States of America, equal to about a fifth (19 per cent) of the world’s total. Germany and the Russian Federation hosted the second and third largest numbers of migrants worldwide (12 million each), followed by Saudi Arabia (10 million)..." 

Now that's interesting data, but like all migration data, you have to read it VERY carefully.  These numbers are total migrants in a country not their own.  In other words, this is the total number of migrants in country X, who might have moved there 20 years ago.  This isn't the number of migrants in any one year.  

However, there IS a data table for migrants in that were hosted by a country in 2000 and 2015 farther down in the UN report.  This table shows the number of migrants in both years.  The net immigration rate is (roughly) the difference between 2015 and 2000 divided by 15 (for the US, that's around 787K / year, which is in the ballpark with other figures we've seen).   

Page 32 of the UN's International Migration Report, 2015.

Visualizing the data: 

As you might expect, for such a hot topic, there is a wealth of data and visualizations of that data to help understand what's going on.  Of course, each visualization is only as good as its underlying data source.  

As Ramon's query shows: 

     [ migration 2015 visualization ] 

leads to a host of visualizations.  Here are a couple (based on UN migration data).  

From Lucify, showing comet trails of people migrating in Europe (click through to see the real interactive visualization): 

And with a query much like the one above, Regular Reader Hans found a large number of infographics and data visualizations collected all on one page.  See The Refugee Crisis through the eyes of Data Visualization.  That's a great collection, here are four that I liked collaged together: 

These can go on forever, but I found one that I really liked, both for its visualization properties, and the sense of the way the world migrations are interlocked.  This is the visualization from PeopleMov.in by Carlo Zapponi.  (you really have to click through to get the full experience).   Gratefully he provided information about where he got his data.  The main source of the data is the Bilateral Migration Matrix (the latest version is 2013) from the World Bank site which is a giant spreadsheet showing movement of people from country X to country Y.  

A note of interpretation:  As we saw above (but worth repeating here), the data set, and the visualization tool shows the estimate of “migrant stocks” in a given country as of 2010.   (“Migrant stocks” are people who live in a country other than the one they were born in, including refugees. Note that these numbers are NOT the number of people crossing the border each year.)  This matrix of data is what's being viewed here (click on any of these to see them full-size):  

This shows the outflow (emigration) from Mexico to other countries.  

If you zoom way out, you can see emigrants from Mexico mostly end up in the US. 

If you click on the USA bar on the right side, you can see where immigrants to the US come from.
Unsurprisingly, many are from Mexico, but there are many from other countries as well, such as the 1.7M from China and the 1.6M from India.

If you take that data (immigration into the US during 2012), you can see that the majority of immigration into the US is from Mexico.  

And with a little exploration, you can see there are a huge number of people moving from place to place...constantly.  

Search Lessons 

Answering this question took a LOT more time than I'd anticipated.  It's easy to find data, but finding the RIGHT data is tricky.  This experience led me to a few essential search insights: 

1.  Check multiple sources.  In my case, I looked at (and found) data from the UN, the Census Department, the DHS (Dept. Homeland Security), and Pew (which in turn drew from data sources in Mexico).  Luckily, the data were pretty close together, though not in perfect agreement... which is to be expected.  

2. The data is there, but you HAVE to read very carefully.  As we saw above, there's a ton of data out there in various repositories--but the metadata describing the data sets always requires a great deal of careful attention to ensure that you've got the right data set, AND that it's actually what you think it is.  (This probably explains why you can read such wildly varying reports of data in the news--it's easy, in the course of fast-paced research, to grab the first numbers that come across your screen.  Especially for these controversial topics, it's essential to have a validation / double-checking step in place.  

3.  Visualizations are great... but be sure you understand what the data is from, and how it's transformed.  Again, it's easy to mis-interpret what you see.  And remember this basic axiom:  If they don't report on where the underlying data comes from, you can't trust the infographic!  

For Teachers

These kinds of assignments typically require a great deal of fairly sophisticated reading.  As a way of getting students to talk about how/what they found, have different teams presenting their findings (including what queries they used, and what sources the found).  This can be a lively discussion, especially as students start to hear about other kinds of research paths than the one they tried. 

An important part of their search strategy will be getting data from particular resources (e.g., UN, UNHCR, DHS, Census, etc.)  They need to learn how to recognize those names as well.  (Think about it this way--a student who doesn't know about the UN won't think to search for that as a reliable data resource.)  So be sure to teach the names of reliable data sources as well.  

Search on!  

Are we making personal search WORSE?

Can it be that we're making search into a harder problem than it already is?

Could be.

Once upon a time, before Google Docs & Spreadsheets, if I knew I had put together a spreadsheet for some kind of analysis, I knew it was in Excel and it was on my desktop.  Search was simple, easy, and elegant.   

Now, perhaps that same document is a Google Spreadsheet OR an Excel file.  This is also true for just plain documents—it was either a .TXT file, a .DOC file (both on my desktop), but now maybe it’s a Google Docs.  OR it's a .DOC.  Or it's on my desktop, or in my cloud storage.  But under which login?

Uh oh.

You can see where this is heading—as tech companies continue to innovate, the number of places and kinds of stuff I have is just going to get more and more complicated.

Let’s take a slightly more realistic problem.  

Suppose you and I are planning a progressive dinner party together. You know, the kind of party where you start at one person's house, then walk to the next house for the next course, etc.  This party will take a bit of planning, so we start putting a few notes together.  But where are the notes kept?

Is it a .DOC file that we email back and forth?  Is it just an email thread? Maybe the notes are kept in the Details of a shared Google Calendar entry on the party date.  Or perhaps you started with a Google Spreadsheet to keep track of all the people and places involved, while I started up a “My Map” on Google Maps to plan a path for the partygoers.  Maybe you bookmarked the web site of a fabulous caterer.  Or to make things worse, maybe I used my work Gmail account to start the thread about the party, but at some point we switched the discussion thread to our personal Gmail accounts.

Dang.

Think about it.  You now have ways to create some kind of note or document… and you can’t really find your notes among all of them.  And this problem is getting worse.  Here’s MY list of places and kinds of documents where I keep notes of different kinds.

> plain txt file on desktop> MS Word> MS Excel> MS Powerpoint> Google Calendar “Details”> Google Docs> Google Spreadsheet> Google Maps “My Map”> Google Bookmarks> browser bookmarks> Google Keep notes> Google PDF files in Drive> Google Presentation> Gmail (work), content in message body> Gmail (work), content in attachment> Gmail (personal) content in message body> Gmail (personal), content in attachment> Earthlink (personal backup email account), both body and attachments> Facebook comment thread, status updates, etc.
> Twitter tweets> Google Chats> Google Site (my personal web site)> Google Blog (my work blog, of which I own 3)
> Google Task list> my non-work, personal blog (and the comments thereon)

(Are there others I’m just not remembering in the heat of writing this essay??)

So now I have to not just remember that I have some kind of note, but also in which format it’s in, which content-containing system it’s in, how to search that system, and the particular limits and properties of that system.

Thank heavens for search!

Except… 

Oh that’s right, no one search tool unites them all.  Each is separate.  Each is different.

What’s worse, almost all of these systems have really different search properties.  I can find a substring in my MS Word document (on the desktop) by using my Desktop search tool, but I can’t find a substring in my blog postings because they're kept in the cloud.  I can’t even search all of my Google Documents for substrings.  (Example:  Suppose I can't remember how to spell Wojcicki?  I can't search for just "Woj"! That kind of search works within a doc, but when you search in Google Drive across all of your documents, it's only search-for-entire-token, no substring search allowed. "Wojcicki" works, "Woj" doesn't.)

Faced with this immense wealth (and complexity) of places to stash notes, I feel a bit like a squirrel looking for my cache of winter nuts in an infinite forest without a map.  Which tree has my cache?  You mean I really have to check each and every tree in the forest?  

Of course the tech companies (and I include myself in this cohort) continue to create ever more ways to store our stuff.  

I heard about a new project to let you create a maps-based itinerary of your trip, with every interesting site and restaurant noted on the path.  Great idea, until I realize that this will be yet-another-place to put notes.  Another place that I’ll need to remember, another place with a disintegrated search system.  Great.

And if you have more than one email account then God Help You.  You’re doomed to searching manually multiple times in multiple places.  (Alas, I have 4, for various legal reasons I can't merge them all together.)

But people cope by segmenting their lives, developing a practice that works for them, putting notes in places where they “naturally” go.  I have all of my personal notes and presentations on my desktop in a plain text file.  And I only ever put notes about upcoming events into my online calendar.  I have one for work, and one for personal events, which means I only have to do two searches…

The rub comes when something slightly new happens, and you don’t already have a worked-out plan for where-to-put-this-note.  And the problem I see increasingly is “I read this somewhere online, but I don’t remember where…”  If you can’t recall some feature that lets you hone in on a particular subsystem, some glimmer of an idea about what kind of thing it was stored in, then you’re in for a long period of looking around.

Question to ask: How many more electronic cubbyholes and clever new document kinds can you mentally support?  How many more should we be making?  It’s clear what my strategy needs to be—I just can’t take on too many more kinds of notes and places to lose my information.

A great policy for our online information supplier to support might be “no new personal information content models without integrated search.”   (I need a snappier catchphrase than that, but you get my point.)

This is an opportunity.  Let’s not continue to make the personal information search problem harder. Let's build an integrated search for personal content AND let's figure out a way to stop producing ever more stuff that's not part of that integrated search story.

Or we’re doomed to be forever searching for our own stuff. 

Search Challenge (3/2/16): Finding out about a concept (part 2)

Let’s continue this "new concept" theme…. 

… of how to get to the core concept behind a glob of text.  

Last week we were looking for “compound concept” terms like "beach music," or "summer romance."  

But another version of this problem is when you know the term, and you know a little bit about the concept, but you really don’t know anything else about what the word means in that context. 

For instance, the word “level” is very common, but in computer game play, it means the entire space (or "level") available to a player while trying to accomplish a given goal (e.g., find the ruby in a complicated maze--that particular maze is the game level).  It also refers to the “level of difficulty” of a given game stage or phase.  

This is an incredibly useful skill as you read.  Often you’ll see words in your text that you can’t quite figure out from context (especially when you’re reading something in a field in which you’re NOT an expert), so this is a great way to learn how to figure out those complex, hidden, subtle meanings.  

Can you figure out how to pin down the definitions of these terms?  Can you give a succinct definition?  (In this sequence, #1 is easy, but #3 is harder.) 

1.  What is an “object” in computer programming?  

2. What is a “model” when used with a bunch of equations to provide some explanatory structure? 

3. In a book I just read, the author wrote, “Miles really knew how to jam in all those modes…”   What’s a “mode”?  (Don't bother to look for this quote--I've modified it so you can't figure it out that way....)  

Can you understand what these terms are all about?  

If so, HOW did you figure out the meanings?  Tell us in the comments! 

Search on! 

Answer: How to find compound concepts

Really well done, SRS team! 

This Challenge was fascinating.  There were more comments in the stream than usual (the ordinary run-rate is around 7 comments / week), all of which are great.  What's more, you found some solutions that I hadn't anticipated--which I always think of as a great outcome.  It tells me that the question is richer and more interesting than I'd thought.  

Let's jump into the answers. 

1.  A friend's child came down with a rare disease that involved an extended period of high fever.  She told me what it was, but I forgot the exact name.  Can you help me find it?  All I remember is that it's called_______ Disease, and the first word is a Japanese name that begins with either an "H" or a "K."  What IS the name of this disease?   

Hans, Luís, and Claire all did a smart thing:  They used the facts given in the description (rare disease, high fever, childhood), did a search, and scanned the snippets for Japanese-seeming names starting with H or K.  

Their queries were some version of: 

      [ child extended period high fever disease ] 

A quick scan of the snippets quickly shows us "Kawasaki Disease" in a couple of the snippet. 

A confirmation search of [ Kawasaki disease ] gives us the following Knowledge Panel, which has a great age-range graph at the bottom. 

2.  I remember reading a book awhile back, that was something like  _________  Oranges.  All I remember about the forgotten term is that it's the name of someone out of Moby Dick (it's like "Ahab," but that's not it).  What IS the name of this book?  

In this case, the simplest search is to search in Google Books and to use a fill-in-the-blank search, like: 

      [ "* "Oranges" " ] 

This query looks odd with an extra set of quote (marked by the extra red quotes), but here's what's going on.  

I started this Challenge by searching for a phrase with an asterisk match like this (that is, without the quotes):   

     [ "* Oranges" ] 

But I noticed that there were a few not quite useful hits in the Search Results Page (SERP). See this odd result at the bottom of this list? 

What's going on here?  

In this case, the book (The Epic Adventures of Lydia Bennet) has enough hits that match the query, [ "* Oranges"].  What could be happening is that synonyms could be registering as well.  So... I put an additional set of quotes around the term "Oranges" to ensure that ONLY phrases with ONLY the word "oranges" would match. In other words, I'm quoting a single term inside of a quoted phrase. 

Hence, this search now shows: 

In this case, I recognized Ishmael as a character from Moby Dick, but if you didn't recall all of the characters, a quick search for: 

     [ list of characters in Moby Dick ] 

will show you a short list of some of the best-known characters in Moby Dick at the top of the page (but be careful:  this is NOT guaranteed to be a complete list).  On the other hand, the Wikipedia entry does look to be pretty complete.  If you scan through this list and then return to the Books search, you'll quickly find the book by Claire Hajaj called Ishmael's Oranges.  

3.  Somewhere in Europe there's a region of the Alps that has an odd, very distinctive name.  It's something like  M________ Alps.  (That is, the first term starts with an "M.")  It's not Mont Blanc, or anything like that.  It's just a single word that starts with "M." And as I remember, it's a kind of odd term to associate with the mountains.    What's the full name of this Alpine region?  

As Ramon pointed out, using an online dictionary service like OneLook (which has a strong partial match capability) is really effective.  

In this case, I just went immediately to OneLook with the query: 

     [ M* Alps ] 

And here's what you see: 

 You can see there are three plausible hits ("Maritime Alps," "Minami Alps," and "Mürzsteg Alps").  A quick couple of clicks tells us that the Minami Alps can be found in Japan, but both the Maritime AND Mürzsteg Alps exist in Central Europe.  

The point of this Challenge is really to bring out methods for doing these kinds of otherwise difficult searches.  I think we've done that... 

Search Lessons  

1. Searching for general topics + visual scanning can be effective.  As we see in #1, the search for the specific topic (an unusual childhood disease), with a quick visual scanning step at the end while looking at the SERP can be effective--especially for properties that are difficult to specify (e.g. a "Japanese-sounding name that starts with H or K").  

2.  The * operator can be useful, especially in constrained searches.  Looking for a book entitled +Oranges can be crazy-making, but is fairly simple if you (a) look in the books collection, (b) use a pattern to specify the compound concept you seek, and (c) use the double-quote to prevent any synonym matching!  

3.  Sometimes the best search for a compound concept is in a computationally enhanced dictionary!  In the Alps example, I turned to the OneLook dictionary and specified a pattern, looking for anything that matched--which includes all (most!) of the mountain ranges of the world.  

For Teachers 

If you're a teacher who's creating as assignment that would use one of these methods, my usual "search lesson" caution applies:  pre-test everything!  

It's also worth exploring the space of possible search options for the online dictionaries.  OneLook's capabilities are fairly extensive--it's not web search, but search in a dictionary, which can be really powerful... especially when you're searching for specific terms / specific concepts.  (For instance, remember our discussion about the "Egg of Columbus"?  Here's the query on OneLook [ "egg of * "] )  

Some nice questions that link concepts to queries might be: 
(1) what kind of choice is a forced choice?  (What name is associated with that?),  
(2) what's the hypothetical universal solvent that was sought by alchemists?
(3) you once went to a place that's called "Big " in Texas.  What is the name of that place?  


Search on!  



(P.S.  I REALLY  haven't forgotten about the immigration / emmigration data Challenge from last week.  I'll write it up today and post my answer tomorrow. Really really really.)