Context-based smartphone sensor logic

• US 9,196,028 B2
• Filed: 09/07/2012
• Issued: 11/24/2015
• Est. Priority Date: 09/23/2011
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rodriguez, Tony F., Bai, Yang
• Primary Examiner(s): NGUYEN, DINH
• Application Number:
  US13/607,095
• Publication Number:
  US 20130150117A1
• Time in Patent Office: 1,173 Days
• Field of Search: 379419-440, 455 901- 903, 455/73, 455/403, 455/550.1
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06V 40/12 Fingerprints or palmprints
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/70 Multimodal biometrics, e.g. combining information from different biometric modalities
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/285 Memory allocation or algorithm optimisation to reduce hardware requirements
• CPC Class Codes: G10L 19/00 Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis in musical instruments G10H
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10

Abstract

Methods employ sensors in portable devices (e.g., smartphones) both to sense content information (e.g., audio and imagery) and context information. Device processing is desirably dependent on both. For example, some embodiments activate certain processor intensive operations (e.g., content recognition) based on classification of sensed content and context. The context can control the location where information produced from such operations is stored, or control an alert signal indicating, e.g., that sensed speech is being transcribed. Some arrangements post sensor data collected by one device to a cloud repository, for access and processing by other devices. Multiple devices can collaborate in collecting and processing data, to exploit advantages each may have (e.g., in location, processing ability, social network resources, etc.). A great many other features and arrangements are also detailed.

Claims

1. A portable user device including an image or sound sensor, a processor, and a memory, the memory containing software instructions causing the device to perform a method that includes:

applying a first classification procedure to information received by said image or sound sensor, to identify a type of said received image or sound information from among plural possible types;

applying a second classification procedure to received second information, to identify a scenario, from among plural possible scenarios, the identified scenario comprising the confluence of at least three circumstances, the received second information being different than said received image or sound information; and

activating one or more of plural recognition modules based on outputs from the first and second classification procedures, and not activating one or more other of said plural recognition modules;

wherein the identified scenario comprises the confluence of at least three circumstances selected from the list consisting of;

information from a first camera of said portable user device, information from a second camera of said portable user device, social networking information, and information about a person in proximity to the user.

2. The device of claim 1 in which the three circumstances are characterized by factors selected from a list consisting of:

time information, location information, ambient light information, information from a first camera of a user portable device, information from a second camera of the user portable device, accelerometer information, alarm information, calendar information, social network information, and information about a person in proximity to the user.

3. The device of claim 2 in which the identified scenario comprises the confluence of at least five circumstances selected from said list.

4. The device of claim 3 in which the identified scenario comprises the confluence of at least seven circumstances selected from said list.

5. A method involving plural different recognition modules and a portable user device, the method comprising the acts:

applying a first classification procedure to received audio and/or visual information, to identify a type of said received audio and/or visual information from among plural possible types;

applying a second classification procedure to received second information, to identify a scenario, from among plural possible scenarios, the identified scenario comprising the confluence of at least three circumstances, the received second information being different than said received audio and/or visual information; and

activating one or more of said recognition modules based on outputs from the first and second classification procedures, and not activating one or more other of said recognition modules;

wherein at least one of said acts is performed by hardware in said portable user device configured to perform such act(s); and

wherein the identified scenario comprises the confluence of at least three circumstances selected from the list consisting of;

information from a first camera of said portable user device, information from a second camera of said portable user device, social networking information, and information about a person in proximity to the user.

6. The method of claim 5 in which the first and second classification procedures identify a temporal-geographic-auditory scenario by reference to time, place, and audio information, and the method includes activating said one or more recognition modules, and not activating said other recognition modules, based on said identified temporal-geographic-auditory scenario.

7. The method of claim 5 that includes activating plural different recognition modules based on outputs from the first and second classification procedures.

8. The method of claim 5 that includes applying outputs from the first and second classification procedures to a rules module, which identifies one or more recognition modules to activate.

9. The method of claim 5 in which the received second information comprises information about one or more persons who are nearby.

10. A method for improving the functioning of a battery-powered portable device, said device including a camera and plural different recognition modules, the method comprising the acts:

applying a classification procedure to received visual information, to identify a type of said received visual information from among plural possible types;

identifying a scenario, from among plural possible scenarios, the identified scenario being based on received second information and comprising the confluence of at least three circumstances, the received second information being different than said received visual information; and

activating one or more of said plural recognition modules based both on said identified type of received visual information and on said identified scenario, and not activating one or more other of said plural recognition modules;

wherein at least one of said acts is performed by hardware configured to perform such act(s); and

wherein battery power is conserved, and execution of other device operations is aided, by not attempting operations that are unneeded or inappropriate in view said identified scenario.

11. The method of claim 10 in which said activating includes activating one or more of said plural recognition modules to process said received visual information, and not activating one or more other of said plural recognition modules to process said received visual information.

12. The method of claim 11 in which the device includes at least two visual recognition modules drawn from the list consisting of:

a facial recognition module, a barcode recognition module, an optical character recognition module, an image fingerprinting module, and an image watermark decoding module, and said activating includes activating one or more of said listed visual recognition modules, and not activating one or more other of said listed visual recognition modules.

13. The method of claim 12 that includes activating plural of said listed visual recognition modules to process the received visual information.

14. The method of claim 12 that includes not activating plural of said listed visual recognition modules to process the received visual information.

15. The method of claim 12 that includes activating the image watermark decoding module to process said visual information, and not activating the image fingerprinting module to process said visual information.

16. The method of claim 12 that includes activating the image fingerprinting module to process said visual information, and not activating the image watermark decoding module to process said visual information.

17. The method of claim 10 that includes identifying a temporal- geographic-imagery scenario by reference to time, place, and image information, and the method includes activating said one or more recognition modules, and not activating said other recognition modules, based on said identified temporal-geographic-imagery scenario.

18. The method of claim 10 that includes applying data, identifying said identified type of received visual information and said identified scenario, to a rules module, and activating one or more of said recognition modules based on control signals output from said rules module.

19. The method of claim 18 that further includes controlling one or more operational parameters of an activated recognition module in accordance with said control signals.

20. The method of claim 19 that includes controlling a frequency at which the activated recognition module performs its recognition function.

21. The method of claim 10 in which the received second information includes information output from said classification procedure.

22. The method of claim 10 in which the received second information includes social network information.

23. The method of claim 10 in which the received second information includes information about one or more persons who are nearby.

24. The method of claim 10 in which the at least three circumstances are characterized by plural factors selected from a list consisting of:

time information, location information, ambient light information, information from a first camera of a user portable device, information from a second camera of the user portable device, accelerometer information, alarm information, calendar information, social network information, and information about a person in proximity to the user.

25. The method of claim 24 in which the identified scenario comprises the confluence of at least three circumstances selected from said list.

26. The method of claim 10 that includes activating an image fingerprint-based recognition module based both on said identified type of received visual information and on said identified scenario.

27. The method of claim 10 that includes activating an image watermark decoding module based both on said identified type of received visual information and on said identified scenario.

28. The method of claim 10 that includes capturing said received visual information with said device camera.

29. A portable device comprising:

an image sensor;

a processor;

a battery;

plural different recognition modules; and

hardware defining software instructions, which software instructions configure the device to perform acts including;

applying a classification procedure to received visual information, to identify a type of said received visual information from among plural possible types;

identifying a scenario, from among plural possible scenarios, the identified scenario being based on received second information and comprising the confluence of at least three circumstances, the received second information being different than said received visual information; and

activating one or more of said plural recognition modules based both on said identified type of received visual information and on said identified scenario, and not activating one or more other of said plural recognition modules;

wherein said device saves battery power, and aids execution of other device operations, by not attempting operations that are unneeded or inappropriate in view of said identified scenario.