SENSOR-BASED MOBILE SEARCH, RELATED METHODS AND SYSTEMS

• US 20140071272A1
• Filed: 07/16/2013
• Published: 03/13/2014
• Est. Priority Date: 10/28/2009
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rodriguez, Tony F., Conwell, William Y.
• Granted Patent: US 9,557,162 B2
• Time in Patent Office: Days
• US Class Current: 348/143
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01C 21/3629 Guidance using speech or audio output, e.g. text-to-speech text to speech systems per se G10L13/00
• CPC Class Codes: G06F 18/24765 Rule-based classification
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06V 10/765 using rules for classification or partitioning the feature space
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

A smart phone senses audio, imagery, and/or other stimulus from a user'"'"'s environment, and acts autonomously to fulfill inferred or anticipated user desires. In one aspect, the detailed technology concerns phone-based cognition of a scene viewed by the phone'"'"'s camera. The image processing tasks applied to the scene can be selected from among various alternatives by reference to resource costs, resource constraints, other stimulus information (e.g., audio), task substitutability, etc. The phone can apply more or less resources to an image processing task depending on how successfully the task is proceeding, or based on the user'"'"'s apparent interest in the task. In some arrangements, data may be referred to the cloud for analysis, or for gleaning. Cognition, and identification of appropriate device response(s), can be aided by collateral information, such as context. A great number of other features and arrangements are also detailed.

Claims

1-27. -27. (canceled)

28. A method comprising:

receiving first data corresponding to imagery captured by a camera of a smartphone, the imagery depicting a subject;

deriving recognition features from the imagery, said deriving being performed by a processing system in the smartphone configured to perform such act;

receiving second data corresponding to non-image stimulus captured by a sensor of the smartphone, said non-image stimulus comprising audio, temperature, magnetic field, smell, or chemical presence;

from a set of reference recognition features associated with a first set of visual subjects, identifying a smaller subset of recognition features associated with a second, smaller set of visual subjects, said identifying being based, at least in part, on the second data; and

identifying the subject from among said second set of subjects, by correspondence between the derived recognition features and recognition features in said subset.

29. The method of claim 28 wherein said recognition features comprise SIFT features.

30. The method of claim 28 wherein said non-image stimulus comprises non-speech audio.

31. The method of claim 28 wherein said non-image stimulus comprises audio from a source different than said subject.

32. The method of claim 28 wherein said non-image stimulus comprises temperature.

33. The method of claim 28 wherein said non-image stimulus comprises magnetic field.

34. The method of claim 28 wherein said non-image stimulus comprises smell.

35. The method of claim 28 wherein said non-image stimulus comprises chemical presence.

36. A method comprising:

receiving first data and second data, one of said data corresponding to imagery captured by a camera of a smartphone, said imagery depicting a subject, the other of said data corresponding to non-image stimulus captured by a sensor of the smartphone, said non-image stimulus comprising audio, temperature, magnetic field, smell, or chemical presence;

deriving recognition features from the first data, said deriving being performed by a processing system in the smartphone configured to perform such act;

selecting a set of reference data based, at least in part, on the second data; and

identifying the first data by correspondence between the derived recognition features and recognition features in said selected set of reference data.

38. The method of claim 36 in which:

the first data comprises image data based on said imagery captured by the smartphone camera;

the second data comprises said non-image stimulus data; and

the method includes;

selecting said set of reference data based at least in part on the non-image stimulus data; and

identifying the subject of said imagery by correspondence between the derived recognition features and recognition features in said selected set of reference data.

39. The method of claim 36 in which:

the first data comprises said non-image stimulus data;

the second data comprises image data based on said imagery captured by the smartphone camera; and

the method includes;

selecting said set of reference data based at least in part on the image data; and

identifying the non-image stimulus by correspondence between the derived recognition features and recognition features in said selected set of reference data.

40. The method of claim 39 in which the non-image stimulus comprises non-speech audio.

37. (canceled)

41. A non-transitory computer readable medium containing software instructions that enable a smartphone configured thereby to perform actions including:

receiving first data and second data, one of said data corresponding to imagery captured by a camera of the smartphone, said imagery depicting a subject, the other of said data corresponding to non-image stimulus captured by a sensor of the smartphone, said non-image stimulus comprising audio, temperature, magnetic field, smell, or chemical presence;

deriving recognition features from the first data;

selecting a set of reference data based, at least in part, on the second data; and

identifying the first data by correspondence between the derived recognition features and recognition features in said selected set of reference data.