Methods and Systems for Controlling an Electronic Device in Response to Detected Social Cues

• US 20170161016A1
• Filed: 12/07/2015
• Published: 06/08/2017
• Est. Priority Date: 12/07/2015
• Status: Active Application

• Current Assignee: Motorola Mobility LLC (Lenovo Group Ltd.)
• Original Assignee: Motorola Mobility LLC (Lenovo Group Ltd.)
• Inventors: Thiagarajan, Balaji, McDunn, Kevin J.
• Granted Patent: US 10,289,381 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

An electronic device includes an audio capture device, one or more physical sensors or context sensors, and one or more processors. A method of operating the electronic device includes receiving audio input, identifying a device command from the audio input, and detecting one or more multi-modal social cues from an environment about the electronic device. Where the one or more multi-modal social cues match one or more predefined criteria, the method executes a control operation in response to the device command.

Claims

1. A method in an electronic device, the method comprising:

receiving, with an audio capture device, audio input;

identifying, with one or more processors, a device command from the audio input;

determining, with a voice interface engine, an identity of a user from the audio input;

detecting, with one or more sensors, one or more multi-modal social cues from an environment about the electronic device; and

where the one or more multi-modal social cues match one or more predefined criteria and the identity of the user is determined, executing, with the one or more processors, a control operation in response to the device command.

2. The method of claim 1, wherein the one or more multi-modal social cues comprise one or more of a physical gesture or a gaze toward the electronic device.

3. The method of claim 1, wherein the one or more multi-modal social cues comprise information relating to an emotional state of one or more persons in the environment.

4. The method of claim 1, wherein the one or more multi-modal social cues comprise:

electronically identifying a companion device to the electronic device; and

receiving an electronic communication indicting the companion device has detected a gesture.

5. The method of claim 2, wherein the one or more multi-modal social cues comprise the gaze toward the electronic device, wherein the detecting the gaze comprises detecting a gaze cone corresponding to a detected gaze direction and determining whether the electronic device is within the gaze cone.

6. The method of claim 5, the executing occurring only where the identity of the user matches a predefined criterion.

7. The method of claim 1, further comprising also receiving one or more additional user inputs.

8. The method of claim 7, where the one or more multi-modal social cues fail to match the one or more predefined criteria, and where the one or more additional user inputs are received, still executing, with the one or more processors, the control operation in response to the device command.

9. A method in an electronic device, comprising:

receiving, with an audio capture device, audio input comprising a device command;

determining, with a voice interface engine operable with the audio capture device, whether a predefined user can be identified from the audio input;

where the predefined user can be identified, actuating, with one or more processors, an intelligent imager;

receiving, with the intelligent imager, one or more multi-modal social cues;

determining, with the one or more processors, whether the one or more multi-modal social cues match one or more predefined criteria; and

where the one or more multi-modal social cues match the one or more predefined criteria, executing, with the one or more processors, a control operation in response to the device command.

10. The method of claim 9, further comprising delivering, with an output device, output to confirm the device command.

11. The method of claim 10, further comprising also receiving, with the audio capture device, a confirmation of the device command.

12. The method of claim 9, further comprising determining, with one or more proximity sensors, how many people are within an environment about the electronic device.

13. The method of claim 12, the executing occurring only where the predefined user is the only person about the electronic device.

14. The method of claim 12, further comprising receiving additional user input and, where the one or more multi-modal social cues fail to match the one or more predefined criteria, still executing the control operation in response to the device command.

15. The method of claim 9, the one or more multi-modal social cues comprising one or more of:

a gaze toward the electronic device;

a gesture toward the electronic device;

or

an image of the predefined user comprising mood information corresponding to the predefined user.

16. An electronic device, comprising:

one or more processors;

an audio capture device, operable with the one or more processors; and

one or more sensors, operable with the one or more processors, the one or more sensors cascaded in a predefined order to detect a plurality of multi-modal social cues to determine whether a device command is intended for the electronic device;

the one or more processors to;

cause the audio capture device to receive an audio input;

identify, from signals from the audio capture device, the device command from the audio input;

detect, with the one or more sensors, one or more multi-modal social cues from an environment about the electronic device; and

where the one or more multi-modal social cues match one or more predefined criteria, execute a control operation in response to the device command.

17. The electronic device of claim 16, the one or more sensors comprising an intelligent imager to detect a gaze toward the electronic device.

18. The electronic device of claim 16, the one or more processors to:

determine whether the device command is a first device command or a second device command from the one or more multi-modal social cues; and

where the device command is the first device command, execute the control operation; and

where the device command is the second device command, execute a different control operation.

19. The electronic device of claim 16, the one or more processors further to distinguish, with the one or more sensors, whether the device command is directed at the electronic device as a function of the one or more multi-modal social cues, and to execute the control operation only when the device command is directed to the electronic device.

20. The electronic device of claim 17, further comprising:

determining an identity of a user from the audio input;

where the user and the device command are identified, delivering an audio prompt requesting confirmation that the device command should be executed;

receiving additional audio input;

identifying, from signals from the audio capture device, a confirmation command from the audio input; and

executing the device command only upon identifying;

the user;

the device command; and

the confirmation command.