Context-based smartphone sensor logic

• US 10,510,349 B2
• Filed: 01/30/2019
• Issued: 12/17/2019
• Est. Priority Date: 04/04/2011
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Rodriguez, Tony F., Bai, Yang
• Primary Examiner(s): Nguyen, Dinh
• Application Number:
  US16/262,634
• Publication Number:
  US 20190237082A1
• Time in Patent Office: 321 Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06V 40/12 Fingerprints or palmprints
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/70 Multimodal biometrics, e.g. combining information from different biometric modalities
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/285 Memory allocation or algorithm optimisation to reduce hardware requirements
• CPC Class Codes: G10L 19/00 Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis in musical instruments G10H
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10

Abstract

Methods employ sensors in portable devices (e.g., smartphones) both to sense content information (e.g., audio and imagery) and context information. Device processing is desirably dependent on both. For example, some embodiments activate certain processor intensive operations (e.g., content recognition) based on classification of sensed content and context. The context can control the location where information produced from such operations is stored, or control an alert signal indicating, e.g., that sensed speech is being transcribed. Some arrangements post sensor data collected by one device to a cloud repository, for access and processing by other devices. Multiple devices can collaborate in collecting and processing data, to exploit advantages each may have (e.g., in location, processing ability, social network resources, etc.). A great many other features and arrangements are also detailed.

Claims

1. A portable user device comprising a processor, a memory and plural sensors, including at least a microphone or camera, the memory containing software instructions configuring the device to perform acts including:

(a) applying a classification procedure to received audio and/or visual information, sensed by the microphone and/or camera, to determine a type of said information from among plural possible types;

(b) determining a scenario type, based at least in part on (i) a number of people sensed as being present with a user of the device, or (ii) an identity of people sensed as being present with said user, together with one or more of;

(i) time of day, (ii) day of week, (iii) location, (iv) calendar data, (v) clock alarm status, (vi) motion sensor data, (vii) orientation sensor data, and (viii) information from a social networking service; and

(c) based on a combination of (i) the determined type of the received audio and/or visual information, and (ii) the determined type of scenario, selecting a group of one or more recognition technologies from a set of available recognition technologies, and applying the selected group of one or more recognition technologies to the received audio and/or visual information;

wherein configuration of said device by said software instructions enables the device, in act (c), to select and apply three or more different groups of recognition technologies to the received audio and/or visual information, in accordance with which of three or more different combinations of (i) information type, and (ii) scenario type, are respectively determined; and

configuration of said device by said software instructions enables the device, in act (c), to include two or more recognition technologies in one of said selected and applied groups of recognition technologies,

at least one of said two or more selected and applied recognition technologies being a watermark-, fingerprint-, barcode-based, or optical character-recognition technology.

2. The device of claim 1, configured by the software instructions to apply said classification procedure to audio information.

3. The device of claim 1, configured by the software instructions to apply said classification procedure to visual information.

4. The device of claim 1, configured by the software instructions to determine a scenario type based at least on said number of people sensed as being present with the user of the device.

5. The device of claim 4, configured by the software instructions to sense the number of people present with the user by reference to Bluetooth signals.

6. The device of claim 4, configured by the software instructions to sense the number of people present with the user by reference to user calendar data.

7. The device of claim 4, configured by the software instructions to sense the number of people present with the user by reference to user location data.

8. The device of claim 4, configured by the software instructions to sense the number of people present with the user by reference to audio signals from an acoustic emitter.

9. The device of claim 1, configured by the software instructions to determine a scenario type based at least on said identity of one or more people sensed as being present with said user.

10. The device of claim 9, configured by the software instructions to sense the identity of one or more people present with the user by reference to Bluetooth signals.

11. The device of claim 9, configured by the software instructions to sense the identity of one or more people present with the user by reference to user calendar data.

12. The device of claim 9, configured by the software instructions to sense the identity of one or more people present with the user by reference to user location data.

13. The device of claim 9, configured by the software instructions to sense the identity of one or more people present with the user by reference to audio signals from an acoustic emitter.

14. The device of claim 1 in which configuration of said device by said software instructions enables the device to include two or more recognition technologies in one of said selected and applied groups of recognition technologies, said two or more recognition technologies including at least one of a watermark-, fingerprint-, barcode-based, or optical character-recognition technology.

15. The device of claim 1, in which configuration of said device by said software instructions enables the device to select and apply a group of one or more recognition technologies including a watermark recognition technology, based on said combination of the determined type of received audio and/or visual information, and the determined type of scenario.

16. The device of claim 1, in which configuration of said device by said software instructions enables the device to select and apply a group of one or more recognition technologies including a fingerprint recognition technology, based on said combination of the determined type of received audio and/or visual information, and the determined type of scenario.

17. The device of claim 1, in which configuration of said device by said software instructions enables the device to select and apply a group of one or more recognition technologies including a barcode-based recognition technology, based on said combination of the determined type of received audio and/or visual information, and the determined type of scenario.

18. The device of claim 1, in which configuration of said device by said software instructions enables the device to select and apply a group of one or more recognition technologies including an optical character recognition technology, based on said combination of the determined type of received audio and/or visual information, and the determined type of scenario.

19. A portable user device comprising a battery, one or more processors, a memory and plural sensors, including at least a microphone or camera, and further comprising:

first means, for applying a classification procedure to received audio and/or visual information, sensed by the microphone and/or camera, to determine a type of said information from among plural possible types, and for producing a corresponding first output;

instructions stored in said memory for configuring the one or more processors to determine a scenario type, and produce a corresponding second output, based at least in part on (i) a number of people sensed as being present with a user of the device, or (ii) an identity of one or more people sensed as being present with said user, together with one or more of;

(i) time of day, (ii) day of week, (iii) location, (iv) calendar data, (v) clock alarm status, (vi) motion sensor data, (vii) orientation sensor data, and (viii) information from a social networking service; and

second means, coupled to receive data from said first and second outputs, for selecting a group of one or more recognition technologies from a set of available recognition technologies, and applying the selected group of one or more recognition technologies to the received audio and/or visual information;

wherein said second means enables the device to select and apply three or more different groups of recognition technologies to the received audio and/or visual information, in accordance with which of three or more different combinations of (i) information type, and (ii) scenario type, are respectively determined; and

wherein said second means enables the device to include two or more recognition technologies in one of said selected and applied groups of recognition technologies, at least one of said two or more selected and applied recognition technologies being a watermark-, fingerprint-, barcode-based, or optical character-recognition technology.

20. A method employing a portable device including a processor, memory and plural sensors that include at least a microphone or a camera, the method comprising the acts:

(a) applying a classification procedure to received audio and/or visual information, sensed by the microphone and/or camera, to determine a type of said information from among plural possible types;

(b) determining a scenario type, based at least in part on (i) a number of people sensed as being present with a user of the device, or (ii) an identity of one or more people sensed as being present with said user, together with one or more of;

(i) time of day, (ii) day of week, (iii) location, (iv) calendar data, (v) clock alarm status, (vi) motion sensor data, (vii) orientation sensor data, and (viii) information from a social networking service; and

(c) based on a combination of (i) the determined type of the received audio and/or visual information, and (ii) the determined type of scenario, selecting a group of one or more recognition technologies from a set of available recognition technologies, and applying the selected group of one or more recognition technologies to the received audio and/or visual information;

at a first time, act (c) comprising applying a first group of recognition technologies to the received audio and/or visual information, in accordance with a first combination of (i) information type, and (ii) scenario type;

at a second time, act (c) comprising applying a second group of recognition technologies to the received audio and/or visual information, in accordance with a second combination of (i) information type, and (ii) scenario type; and

at a third time, act (c) comprising applying a third group of recognition technologies to the received audio and/or visual information, in accordance with a third combination of (i) information type, and (ii) scenario type;

wherein;

said first, second and third groups of recognition technologies are different;

said first, second and third combinations are different; and

at one of said first, second or third times, act (c) includes selecting and applying a group of two or more recognition technologies to the received audio and/or visual information, at least one of said two or more recognition technologies being a watermark-, fingerprint-, barcode-based, or optical character-recognition technology.