Methods and systems for controlling an electronic device in response to detected social cues

• US 10,289,381 B2
• Filed: 12/07/2015
• Issued: 05/14/2019
• Est. Priority Date: 12/07/2015
• Status: Active Grant

• Current Assignee: Motorola Mobility LLC (Lenovo Group Ltd.)
• Original Assignee: Motorola Mobility LLC (Lenovo Group Ltd.)
• Inventors: McDunn, Kevin J, Thiagarajan, Balaji
• Primary Examiner(s): Yang, Qian
• Application Number:
  US14/961,224
• Publication Number:
  US 20170161016A1
• Time in Patent Office: 1,254 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

An electronic device includes an audio capture device, one or more physical sensors or context sensors, and one or more processors. A method of operating the electronic device includes receiving audio input, identifying a device command from the audio input, and detecting one or more multi-modal social cues from an environment about the electronic device. Where the one or more multi-modal social cues match one or more predefined criteria, the method executes a control operation in response to the device command.

Claims

1. A method in an electronic device, the method comprising:

receiving, with an audio capture device, audio input, wherein the audio capture device comprises a microphone in an always ON state;

identifying, with one or more processors, whether a device command is present in the audio input;

determining, with a voice interface engine, an identity of a user from the audio input;

detecting, with one or more sensors, one or more multi-modal social cues from an environment about the electronic device;

where the device command is present in the audio input, corroborating, from the one or more multi-modal social cues, that the device command is intended for the electronic device; and

executing, with the one or more processors, a control operation in response to the device command only where the user is identified as an owner of the electronic device and corroboration from the one or more multi-modal social cues confirms that the device command is intended for the electronic device.

2. The method of claim 1, further comprising actuating a wide-angle camera and capturing an image in response to identifying that the device command is present in the audio input, wherein the one or more multi-modal social cues comprise a gaze toward the electronic device, wherein the corroborating that the device command is intended for the electronic device comprises analyzing the image to determine if the owner of the electronic device is gazing toward the electronic device.

3. The method of claim 2, wherein the one or more multi-modal social cues comprise the gaze toward the electronic device, wherein the detecting the gaze comprises detecting a gaze cone corresponding to a detected gaze direction and determining whether the electronic device is within the gaze cone.

4. The method of claim 3, further comprising further confirming that the user is the owner of the electronic device by capturing an image of the environment of the electronic device in response to identifying that the device command is present in the audio input, analyzing the image to confirm that a face of the owner is present in the image, the executing occurring only where the face of the owner is present in the image.

5. The method of claim 1, further comprising actuating a wide-angle camera and capturing an image in response to identifying that the device command is present in the audio input, wherein the one or more multi-modal social cues comprise a physical gesture, wherein the corroborating that the device command is intended for the electronic device comprises analyzing the image to determine if the owner of the electronic device is pointing a finger toward the electronic device.

6. The method of claim 1, wherein the one or more multi-modal social cues comprise:

electronically identifying a companion device to the electronic device; and

receiving an electronic communication indicting the companion device has detected a gesture, wherein the corroborating that the device command is intended for the electronic device comprises determining, from the companion device, that the owner of the electronic device made the gesture.

7. A method in an electronic device, comprising:

receiving, with an audio capture device, audio input;

determining, with one or more processors, whether the audio input comprises a device command;

determining, with a voice interface engine operable with the audio capture device, whether a predefined user can be identified from the audio input;

where both the audio input comprises the device command and the predefined user is identified, actuating, with one or more processors, an intelligent imager;

receiving, with the intelligent imager, one or more multi-modal social cues and confirming, with the intelligent imager, the predefined user is an owner of the electronic device;

determining, with the one or more processors, whether the one or more multi-modal social cues match one or more predefined criteria; and

where both the one or more multi-modal social cues match the one or more predefined criteria and the predefined user is confirmed as the owner of the electronic device, executing, with the one or more processors, a control operation in response to the device command.

8. The method of claim 7, further comprising delivering, with an output device, output to confirm the device command.

9. The method of claim 8, further comprising also receiving, with the audio capture device, a confirmation of the device command.

10. The method of claim 7, further comprising determining, with one or more proximity sensors, how many people are within an environment about the electronic device.

11. The method of claim 10, the executing occurring only where the predefined user is the only person about the electronic device.

12. The method of claim 10, further comprising receiving additional user input and, where the one or more multi-modal social cues fail to match the one or more predefined criteria, still executing the control operation in response to the device command.

13. The method of claim 7, the one or more multi-modal social cues comprising one or more of:

a gaze toward the electronic device;

a gesture toward the electronic device;

or

an image of the predefined user comprising mood information corresponding to the predefined user.

14. An electronic device, comprising:

one or more processors;

an audio capture device, operable with the one or more processors; and

one or more sensors, operable with the one or more processors, the one or more sensors cascaded in a predefined order to detect a plurality of multi-modal social cues to determine whether a device command is intended for the electronic device;

the one or more processors;

causing the audio capture device to receive an audio input;

identifying, from signals from the audio capture device, the device command from the audio input;

determining an identity of a user from the audio input, and where the user and the device command are identified, delivering an audio prompt requesting confirmation that the device command should be executed;

receiving additional audio input;

identifying, from signals from the audio capture device, a confirmation command from the additional audio input;

detecting, with the one or more sensors, one or more multi-modal social cues from an environment about the electronic device; and

where the one or more multi-modal social cues match one or more predefined criteria, the confirmation command is identified from the additional audio input, and the user is identified from the audio input, executing a control operation in response to the device command;

the one or more processors further distinguishing, with the one or more sensors, whether the device command is directed at the electronic device as a function of the one or more multi-modal social cues, and to execute the control operation only when the device command is directed to the electronic device.

15. The electronic device of claim 14, the one or more sensors comprising an intelligent imager to detect a gaze toward the electronic device.

16. The electronic device of claim 15, wherein the one or more social cues comprise an approach gesture.

17. The electronic device of claim 14, the one or more processors to:

determine whether the device command is a first device command or a second device command from the one or more multi-modal social cues; and

where the device command is the first device command, execute the control operation; and

where the device command is the second device command, execute a different control operation.

18. The electronic device of claim 14, wherein when the additional audio input fails to include the confirmation command, employing, by the one or more processors, the additional audio input to improve accuracy in identifying whether received audio input includes confirmation commands.