Sensor-based mobile search, related methods and systems

• US 8,489,115 B2
• Filed: 05/08/2012
• Issued: 07/16/2013
• Est. Priority Date: 10/28/2009
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rodriguez, Tony F., Conwell, William Y.
• Primary Examiner(s): MIZRAHI, DIANE D
• Application Number:
  US13/466,803
• Publication Number:
  US 20120220311A1
• Time in Patent Office: 434 Days
• Field of Search: 455/456.1, 455/556.1
• US Class Current: 455/456.1
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01C 21/3629 Guidance using speech or audio output, e.g. text-to-speech text to speech systems per se G10L13/00
• CPC Class Codes: G06F 18/24765 Rule-based classification
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06V 10/765 using rules for classification or partitioning the feature space
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user'"'"'s environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone'"'"'s camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user'"'"'s apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

Claims

1. A method comprising:

receiving first data corresponding to imagery captured by a camera of a smartphone, the imagery depicting an unknown subject;

receiving second data corresponding to non-image stimulus captured by a sensor of the smartphone, said non-image stimulus being other than speech;

processing the received first and second data to generate output data indicating a possible identity of said unknown subject, said processing being performed at least in part by a processing system in the smartphone that is configured to perform part or all of said processing;

wherein said processing includes consulting disambiguation information recalled from a data store, the disambiguation data having been earlier obtained by analysis of a corpus of content information posted by plural third parties to one or more public internet sites.

2. The method of claim 1 that additionally includes analyzing said corpus of content to derive the disambiguation data.

3. The method of claim 2 in which the analyzing includes analyzing visual content.

4. The method of claim 2 in which the analyzing includes analyzing textual metadata accompanying visual content.

5. The method of claim 1 in which the sensor comprises a microphone.

6. The method of claim 1 in which the sensor comprises a thermometer.

7. The method of claim 1 in which the sensor comprises a location sensing system.

8. The method of claim 1 in which the sensor comprises a chemical sensor.

9. A method comprising:

receiving first data corresponding to imagery captured by a camera of a smartphone, the imagery depicting an unknown subject;

processing the first data to generate object identification data indicating a possible identity of an object depicted in the imagery, said object being different than said unknown subject; and

recognizing the unknown subject, said recognizing being based at least in part both on said object identification data, and also on co-occurrence information involving said object;

wherein said co-occurrence information was earlier obtained by analysis of a corpus of content information posted by plural third parties to one or more public internet sites.

10. The method of claim 9 that additionally includes analyzing said corpus of content to derive the co-occurrence information.

11. The method of claim 10 in which the analyzing includes analyzing visual content.

12. The method of claim 10 in which the analyzing includes analyzing textual metadata accompanying visual content.

13. A method comprising:

receiving first data corresponding to imagery captured by a camera of a smartphone, the imagery depicting an unknown subject;

receiving second data corresponding to non-speech audio stimulus captured by a microphone of the smartphone;

processing the received first and second data to generate output data indicating a possible identity of said unknown subject, said processing being performed at least in part by a processing system in the smartphone that is configured to perform part or all of said processing;

wherein said processing includes computing a score for each of plural different candidate identifications of said unknown subject, at least one of said scores reflecting an uncertainty factor corresponding to the audio stimulus.

14. A method comprising:

receiving first data corresponding to imagery captured by a camera of a smartphone, the imagery depicting an unknown subject;

deriving recognition features from the imagery, said deriving being performed by a processing system in the smartphone configured to perform said deriving;

receiving second data corresponding to non-image stimulus captured by a sensor of the smartphone, said non-image stimulus comprising audio or temperature;

from a set of reference recognition features associated with a first set of visual subjects, identifying a smaller subset of recognition features associated with a second, smaller set of visual subjects, said identifying being based, at least in part, on the second data; and

identifying the unknown subject from among said second set of subjects, by correspondence between the derived recognition features and recognition features in said subset.

15. The method of claim 14 wherein said recognition features comprise scale invariant features.

16. The method of claim 14 wherein said non-image stimulus comprises non-speech audio.

17. The method of claim 14 wherein said non-image stimulus comprises audio from a source different than said subject.

18. The method of claim 14 wherein said non-image stimulus comprises temperature.

19. A non-transitory computer readable storage medium containing software instructions that, when executed by a processor of a camera-equipped smartphone device, cause the smartphone device to perform acts including:

receiving first data corresponding to imagery captured by the camera of a smartphone, the imagery depicting an unknown subject;

receiving second data corresponding to non-image stimulus captured by a sensor of the smartphone, said non-image stimulus being other than speech;

processing the received first and second data to generate output data indicating a possible identity of said unknown subject;

wherein said processing includes consulting disambiguation information recalled from a data store, the disambiguation data having been earlier obtained by analysis of a corpus of content information posted by plural third parties to one or more public internet sites.