Sensor-based mobile search, related methods and systems

• US 9,557,162 B2
• Filed: 07/16/2013
• Issued: 01/31/2017
• Est. Priority Date: 10/28/2009
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rodriguez, Tony F, Conwell, William Y.
• Primary Examiner(s): HARVEY, DAVID E
• Application Number:
  US13/943,176
• Publication Number:
  US 20140071272A1
• Time in Patent Office: 1,295 Days
• Field of Search: 348/143
• US Class Current: 1/1
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01C 21/3629 Guidance using speech or audio output, e.g. text-to-speech text to speech systems per se G10L13/00
• CPC Class Codes: G06F 18/24765 Rule-based classification
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06V 10/765 using rules for classification or partitioning the feature space
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user'"'"'s environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone'"'"'s camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user'"'"'s apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

Claims

1. A method comprising:

receiving first data corresponding to imagery captured by a camera of a smartphone, the imagery depicting a visual subject;

deriving recognition features from the imagery, said deriving being performed by a processing system in the smartphone configured to perform such act;

receiving second data corresponding to non-image stimulus captured by a sensor of the smartphone, said non-image stimulus comprising at least one stimulus selected from the group consisting of;

audio, temperature, magnetic field, smell, or chemical presence;

from a set of reference recognition features associated with a first set of visual subjects, identifying a smaller subset of recognition features associated with a second, smaller set of visual subjects, said smaller set of visual subjects including first and second visual subjects;

using the non-image stimulus, classifying an environment of the smartphone by assigning a first probability value that the smartphone environment is a first environment and assigning a second probability value that the smartphone environment is a second environment, both of said probability values being more than 0% and less than 100%;

obtaining two values respectively indicating likelihoods of encountering the first visual subject in said first and second environments, and obtaining two other values respectively indicating likelihoods of encountering the second visual subject in said first and second environments; and

combining said probability and likelihood values together in assessing that the visual subject is more likely to be the first visual subject than the second visual subject;

wherein the visual subject is identified from among said second set of subjects, by correspondence between the derived recognition features and recognition features in said subset, and by use of said probability and likelihood values.

2. The method of claim 1 wherein said recognition features comprise SIFT features.

3. The method of claim 1 wherein said non-image stimulus comprises non-speech audio.

4. The method of claim 1 wherein said non-image stimulus comprises audio from a source different than said subject.

5. The method of claim 1 wherein said non-image stimulus comprises temperature.

6. The method of claim 1 wherein said non-image stimulus comprises magnetic field.

7. The method of claim 1 wherein said non-image stimulus comprises smell.

8. The method of claim 1 wherein said non-image stimulus comprises chemical presence.

9. The method of claim 1 that includes identifying the smaller subset of recognition features associated with the second, smaller set of visual subjects based, at least in part, on the second data.

10. A non-transitory computer readable medium containing software instructions operative to configure a processor- and camera-equipped smartphone system to perform acts including:

receiving first data corresponding to imagery captured by the camera, the imagery depicting a visual subject;

deriving recognition features from the imagery;

receiving second data corresponding to non-image stimulus captured by a sensor of the smartphone system, said non-image stimulus comprising at least one stimulus selected from the group consisting of;

audio, temperature, magnetic field, smell, or chemical presence;

from a set of reference recognition features associated with a first set of visual subjects, identifying a smaller subset of recognition features associated with a second, smaller set of visual subjects, said smaller set of visual subjects including first and second visual subjects;

using the non-image stimulus, classifying an environment of the camera by assigning a first probability value that the camera environment is a first environment and assigning a second probability value that the camera environment is a second environment, both of said probability values being more than 0% and less than 100%;

obtaining two values respectively indicating likelihoods of encountering the first visual subject in said first and second environments, and obtaining two other values respectively indicating likelihoods of encountering the second visual subject in said first and second environments; and

combining said probability and likelihood values together in assessing that the visual subject is more likely to be the first visual subject than the second visual subject;

wherein the visual subject is identified from among said second set of subjects, by correspondence between the derived recognition features and recognition features in said subset, and by use of said probability and likelihood values.

11. A smartphone system including the computer readable memory of claim 10, together with one or more processors, a screen, a touch sensor, a camera, and a wireless interface.