Tuesday, October 22, 2013

What recognition-based search apps are there?

When I posed the original question, what I meant to ask was "What recognition-based tools are there that we can use for search?"  

Here's our provisional list so far.  These are all apps (or tools) you can use to recognize objects or signals in the world.  

I've grouped them by sensing mode (that is, how does your query device sense the signal it's going to try and recognize).  When you think about it that way, all of the "image reco" apps go together (they're all variations on image processing) and all of the "audio reco" apps go together.  

What sense modalities are we missing here?  

Obviously, there's no taste reco system (or is there one and I don't know about it??) Wouldn't it be interesting if there were a wine varietal recognition tool.  (Pour a little of your plonk onto your cell phone--have it identify the variety, terrior, and year!)  

Remember--our goal in this is to increase the range of tools we know about that help us be more effective searchers.  

Are there more reco-based search tools that we're missing from this list?  

Search on (for more search tools...)!  


Google Goggles                      Runs augmented Image search of scene.       

LeafSnap                                  Recognizes deciduous leaves of North American trees.
LookTel                                     Money; cans of food. Intended for low vision users to find everyday items.
Evernote                                   Part of Evernote does handwritten text recognition.
Google Language Translate             
                                               Recognizes hand-written Chinese, Japanese, Arabic, etc.
Picasa / iPhoto                         Recos faces from the set of faces you’ve tagged .           

WhatTheFont                            Identifies the font in which a given text is set in an image.
IdMyPill                                   Identifies pills by size,shape, color.               
Project Shepherd                      Identifies individual animals out of a tagged collection by
                                                      their particular spot pattern.  Open source project.
                                                      Requires some setup to use. Works for whale sharks, 
                                                      manta rays, etc.  
                                                  Searches for images based on similarity to images on the net.                                                   
gPick                                          Identifies a specific region of an image by color name
                                                      (e.g., “teal” or “plum”)
PDFpen Scan+
Tesseract (open source)
Pixter                                         Number of character recog tools.  Point at an image of text and recover the
                                                      the text itself by recognition.  (Which you can then search via Control-F.) 
StarMap                                     Augmented Reality apps.  Point your phone at a position in the sky and 
                                                     see what's there. 

 Meal Snap                                Analyzes your meal to determine your caloric / health consequence of 
                                                     eating everything on your plate. 

ArtCapture                                 Recognizes "more than 500K" works of art based on cell camera image. 


Soundhound                             Identify pre-recorded music (not live performance)

WeBird                                    Identify bird songs (Android)                                                     

SongTapper                             Recognize (some) popular songs by tapping the rhythm on the space bar
                                                      (It worked rarely for me…)
Viggle                                      Identifies TV show you’re watching by recognizing the sound track.
Musipedia                               Identify music by whistling (difficult to make work), or playing on
                                                      a keyboard.

TunePal                                   Identify live performances of Irish Session tunes.


GPS / Location sensing

FieldTrip                               Looks for interesting historical sites near your current location
                                                      (US only?)

Rutísima Madre                    Looks for interesting historical sites near your current location
                                                      (Mexico only?)

GeoGoogle                           GPS augmented info.  
Peaks                                  Mountain peak ID by GPS + camera
NokiaCity Lens                     Identify buildings in city by AR on cell phone


  1. I would call Field Trip international. Looking through the categories they have just for architecture and historic events, I see things for Dublin, Historic London, Berlin and more. The cool and unique category includes items from Atlas Obscura that appears to be a worldwide travel database http://www.atlasobscura.com/ .

  2. Although olfactory (smell / odor) recognition appears to be technically less difficult than taste recognition, I don't know of any working e-nose for smartphone.

    Another modality (different from location sensing, but probably best working together) is kinetic (movement and balance) recognition. This might include gesture recognition (movement outside the recognition device or on the touch screen) and proprioception (device movement recognition, using a gyroscope and / or signal location).

    Still another modality would be thermoception (temperature recognition, thermometer). I don't know of any smartphone with this capability. Instead, they use data from a web service after locating you.

    Other modalities can be thought of. Check the Wikipedia article on Sense.

    Then, within image recognition, there's facial recognition.

    Still related to image recognition, before Google Search by Image was any good, I used the excellent TinEye Reverse Image Search a lot. I liked it so much I was sure Google would buy it (only it didn't). :)

  3. don't know if this is still in the pipeline, but would seem to fall in the '"kinetic" category you defined as well as language translation by image/symbol/sign.
    ☞language translator

    1. It definitely fits. Query: [ what is that person saying in ASL ]

  4. Replies
    1. Interesting. The Technabling is a sensor net that detects abnormal patterns of behavior--a neat app, but not "search-y" in the normal sense. Fascinating, tho.

  5. So I went down this rabbit hole [ how many senses do human have ] (I figured I give hummingbird semantic search a try) and got to Humans Have a Lot More Than Five Senses

    Back to searching for apps and/or web sites for reco.

  6. Fred -- It's true--we do have many more than 5 senses. (Likewise, computers have many more senses than we do. For instance, measuring the number of disk errors can be an indicator of earthquakes--see: http://www.eduard-heindl.de/IEEE-p2p-heindl.pdf )

    It would be interesting to figure out what "sense" computers really DO have!

  7. Dan, not to be overly risqué in response, but this is sign language related - as well as art and computer manufactured and tech related and your daughter's art class — yesterday, in Prague… not a text or tweet, but still a bird… of sorts - hope there isn't a test to trace it to the Etruscans.
    message in Prague
    from London 2012
    Wikipedia entry

  8. Kind of off topic.

    Reading my streamline on Google Plus found Handwriting input comes to Gmail and Google Docs It is not an app; and, it is a sequel from the What does this character mean?

    Have a great day. See you tomorrow in a new Search Research Challenge

  9. GPS / Location sensing

    Most of the navigation apps begin with identifying your location first, then helping you find directions to a destination. Most allow you to either type in your destination address or search for it. Like Field Trip where you aren't searching for something specific, these apps can also offer different types of places near your location such as stores, hotels, gas stations and more.

    Three apps that I use regularly on iOS (also available on Android)
    Google Maps Mobile



    Staying with the GPS/Location sensing apps
    LocalScope (iOS) is again similar to Field Trip in that it is a location browser. Based on your location, Localscope can help you find Google+ reviews, Twitter tweets, Flickr photos, Youtube videos and many more. You can view the results in a list, on a map or through the camera in an augmented reality view. Localscope integrates with the 3 apps above and more for directions. Localscope supports 21 languages.

    Similar to Localscope
    AroundMe (iOS, Android, Windows Phone)

    An occasional hobby of mine is Geocaching. For those unfamiliar with geocaching, it is a type of treasure hunt based on GPS location. You are provided coordinates for a hidden cache. Using your GPS device you hunt for cache using your position and the position of the cache. The Geocaching app (iOS, Android, Windows Phone) allows you to find nearby caches based on your location wherever you are. Then as you search for the cache the app constantly updates based on your location. Searching IRL!

  10. Another one is Bar Code Scanner, and other bar code scanning tools

  11. IntoNow is a great audio app that can identify a TV show and episode with just a few seconds of audio.

  12. While looking at optical character recognition apps, I remembered that there are cheating apps for apps like Words With Friends and Hanging With
    Friends. You take a screenshot of your game screen and then uploaded to the cheating app. You are then presented with a list of possible words
    to play. That reminded me of a post that taught me how to cheat at Words With Friends using Wolfram Alpha. The author of the post stated "Wolfram Alpha is a remarkable resource that's chock full of ways to find out
    about science and math."

    Thinking of that post, Wolfram Alpha is recognizing letters and spaces (blanks) and then identifying words that fit the pattern. Example Re_earch. Keywords that we have been using are recognizing and then identifying.

    I wondered if Wolfram Alpha could recognize and then identify other patterns. I found that Wolfram Alpha can recognize and identify number sequences [ 1,1,2,3,5,8,13,...
    ]as well as recognize and identify genealogic relationships [ mother's father's sister's daughter's son ].

    Wolfram Alpha (iOS, Android, nook, Kindle Fire)
    Now I know this gets away from the framework of human senses, it is
    still "a remarkable resource."

  13. Aurasmas - see this video that explains the software www.youtube.com/watch?v=JrMxVmvBAVg

    While it may be used as a marketing tool because anyone can create an "aura" it can be used to bring print to life. I can see all kinds of applications. What I couldn't figure out is how to use it to create an aura using an online object.

    I can understand the 'recognition' app to a certain point. If its geographically related than unless we can simulate the location searching isn't possible. On a Garmin you can simulate the coordinates ( I use it if creating a route for a hike using my Garmin). What would be really cool is if we could use Street View.

    Two other apps that I have very little knowledge of is Layar and Junaio. I'm not sure if these are any better than say Google Earth POI's but they provide a mobile platform.

  14. Here's a link to investigative tools used by Journalists that links to sites we could use so this goes under 'tools'.
    I know we have used it before but another tool that is expanding is the "Layers" in Google Earth. I see they now have a link to historical maps and of course National Geographic but many more.

  15. I may have forgotten to give the link to The Journalist's Toolbox


    1. Thanks Rosemary, The Journalist's Toolbox is a treasure trove - additional suggestions from there:
      Gannett; 14 Best…
      as everything is becoming a mobile space, this might be the quickest way to keep up with the updates…
      JT on twitter
      an indication of how quickly things change; this was a featured site on the J'toolbox box… R.I.P, Scirus -
      scirus, gone in 2014

      oarfish search example:
      from the scirus site - pre note keeping apps -
      new scientist

  16. Google+ Computer Vision Search - began in May and was expanded to more than 1,000 objects this week. Search Google+ photos for terms like [ labrador ] [ food ] and more.

    1. Wondering if the Computer Vision Search is related to NEIL (Never Ending Image Learner)

  17. I saw this post today by Google in Education. If the Smart Measure tool works as advertised then Smart Tools gets added to the list.

  18. Two more:
    Get your next eye exam on a smartphone

    This smartphone app can detect skin cancer - this appears to do what we through the U of M skinscan app did, but requires a $500 lens to do it.

  19. Jetpac offers two apps for iOS.

    Deep Belief allows you to teach your phone to recognize objects.

    Spotter allows your iPhone camera to identify objects. I tired this one and it was able to identify a few things in my office accurately. I haven't figured out how useful this is to the average person. Does anyone have any ideas?

    I can see where the usage data from the teacher app might be used in a project mentioned above such as Google's Computer vision, but on a day to day use I can't think of how this would be helpful.

  20. Store receipt scanning apps seem to be on the rise. A couple of apps I've been using are Snap and Checkout51. They are basically electronic coupon apps. You tell the app what you bought and upload a picture of your receipt. They verify your purchase and credit you money for your purchase.

    A new one I just found today is CookBrite. Unfortunately, it is Android only so I can't try it out, but the concept is very useful. You can keep a record of what you have on hand by snapping a picture of your receipt. You can then search for recipes based on what you have on hand.