Intelligent assistant device communicating non-verbal cues

• US 11,010,601 B2
• Filed: 03/26/2018
• Issued: 05/18/2021
• Est. Priority Date: 02/14/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bathiche, Steven Nabil, Pradeep, Vivek, Bennett, Alexander Norman, O'Neil, Daniel Gordon, Reed, Anthony Christian, Luchowiec, Krzysztof Jan, Kolawole, Tsitsi Isabel
• Primary Examiner(s): Patel, Shardul D
• Application Number:
  US15/936,076
• Publication Number:
  US 20180232571A1
• Time in Patent Office: 1,149 Days
• Field of Search: 701495, 382103, 348148, 434219, 434323, 901 1, 901 47, 700253
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: H05B 47/105 in response to determined parameters
• CPC Class Codes: H05B 47/115 by determining the presence or movement of objects or living beings
• CPC Class Codes: H05B 47/125 by using cameras
• CPC Class Codes: Y02B 20/40 Control techniques providing energy savings, e.g. smart controller or presence detection

Abstract

An intelligent assistant device is configured to communicate non-verbal cues. Image data indicating presence of a human is received from one or more cameras of the device. In response, one or more components of the device are actuated to non-verbally communicate the presence of the human. Data indicating context information of the human is received from one or more of the sensors. Using at least this data one or more contexts of the human are determined, and one or more components of the device are actuated to non-verbally communicate the one or more contexts of the human.

Claims

1. At an intelligent assistant device configured to respond to natural language inputs, a method for communicating non-verbal cues, the method comprising:

from one or more cameras of the intelligent assistant device, receiving image data indicating presence of a human;

in response to receiving the image data, actuating one or more components of the intelligent assistant device to non-verbally communicate the presence of the human;

from one or more sensors of the intelligent assistant device, receiving data indicating context information of the human;

using at least the data indicating context information of the human, determining one or more contexts of the human, wherein the one or more contexts of the human comprise (1) an initial identity of the human that corresponds to a previously-identified human and represents an initial confidence value, and/or (2) a verified identity of the human that represents a verified confidence value greater than the initial confidence value; and

in response to determining the one or more contexts of the human, actuating one or more components of the intelligent assistant device to non-verbally communicate the one or more contexts of the human.

2. The method of claim 1, wherein actuating one or more components of the intelligent assistant device to non-verbally communicate the presence of the human further comprises illuminating at least one light source located on the intelligent assistant device.

3. The method of claim 1, wherein actuating one or more components of the intelligent assistant device to non-verbally communicate the presence of the human further comprises one or more of moving a camera to foveate on the human and moving a display to follow a location of the human.

4. The method of claim 1, wherein the one or more contexts of the human comprise at least one context selected from (1) a location of the human with respect to the intelligent assistant device, (2) a height of the human, and (3) a distance of the human from the intelligent assistant device.

5. The method of claim 1, wherein actuating one or more components to non-verbally communicate the one or more contexts of the human further comprises illuminating at least one light source located on the intelligent assistant device, and illuminating the at least one light source comprises modulating at least one of a frequency, a brightness, a color, a quantity, and a shape of the at least one light source.

6. The method of claim 5, wherein the at least one light source is a plurality of light sources, and the plurality of light sources comprises a plurality of LEDs.

7. The method of claim 1, wherein actuating one or more components to non-verbally communicate the one or more contexts of the human further comprises displaying vector graphics via a display of the intelligent assistant device.

8. The method of claim 1, wherein actuating one or more components to non-verbally communicate the one or more contexts of the human further comprises projecting the non-verbal cues onto a surface.

9. The method of claim 1, wherein the human is a first human, further comprising:

from one or more sensors of the intelligent assistant device, receiving an indication of presence of a second human; and

illuminating at least one light source located on the intelligent assistant device to non-verbally communicate that the intelligent assistant device is responsive to the natural language inputs from the first human.

10. The method of claim 1, wherein the human is a first human and the one or more contexts of the human comprise the location of the first human, the method further comprising:

from one or more sensors of the intelligent assistant device, receiving an indication of presence of a second human; and

illuminating at least one light source located on the intelligent assistant device to non-verbally communicate that the intelligent assistant device is tracking the location of the first human and a location of the second human.

11. The method of claim 1, wherein the one or more contexts of the human comprise the initial identity of the human, the method further comprising receiving and fusing from a plurality of different sensors of the intelligent assistant device the data indicating context information of the human to determine the initial identity of the human.

12. The method of claim 1, further comprising illuminating at least one light source located on the intelligent assistant device to non-verbally communicate a type of sensor data being used by the intelligent assistant device to determine the one or more contexts of the human.

13. The method of claim 1, wherein the one or more components comprise a plurality of light sources, and the plurality of light sources comprises at least one infrared emitter, the method further comprising illuminating at least one light source of the plurality of light sources to one or more of (1) reduce visibility of the at least one infrared emitter, and (2) incorporate light emitted from the at least one infrared emitter into an illumination pattern produced by the at least one light source.

14. An intelligent assistant device configured to respond to natural language inputs, comprising:

a plurality of light sources;

a plurality of sensors comprising one or more cameras;

at least one speaker;

a logic machine; and

a storage machine holding instructions executable by the logic machine to;

from at least one camera of the one or more cameras, receive image data indicating presence of a human;

in response to receiving the image data, actuate one or more components of the intelligent assistant device to non-verbally communicate the presence of the human;

from one or more sensors of the plurality of sensors, receive data indicating context information of the human;

using at least the data indicating context information of the human, determine one or more contexts of the human, wherein the one or more contexts of the human comprise (1) an initial identity of the human that corresponds to a previously-identified human and represents an initial confidence value, and/or (2) a verified identity of the human that represents a verified confidence value greater than the initial confidence value; and

in response to determining the one or more contexts of the human, actuate one or more components of the intelligent assistant device to non-verbally communicate the one or more contexts of the human.

15. The intelligent assistant device of claim 14, wherein actuating one or more components of the intelligent assistant device to non-verbally communicate the presence of the human further comprises illuminating at least one light source located on the intelligent assistant device.

16. The intelligent assistant device of claim 14, wherein actuating one or more components of the intelligent assistant device to non-verbally communicate the presence of the human further comprises one or more of moving a camera to foveate on the human and moving a display to follow a location of the human.

17. The intelligent assistant device of claim 14, wherein actuating one or more components to non-verbally communicate the one or more contexts of the human further comprises illuminating at least one light source located on the intelligent assistant device, and illuminating the at least one light source comprises modulating at least one of a frequency, a brightness, a color, a quantity, and a shape of the at least one light source.

18. The intelligent assistant device of claim 14, wherein the human is a first human, and the instructions are executable to:

from one or more sensors of the intelligent assistant device, receive an indication of presence of a second human; and

illuminate at least one light source located on the intelligent assistant device to non-verbally communicate that the intelligent assistant device is responsive to the natural language inputs from the first human.

19. The intelligent assistant device of claim 14, wherein the human is a first human and the one or more contexts of the human comprise the location of the first human, and the instructions are executable to:

from one or more sensors of the intelligent assistant device, receive an indication of presence of a second human; and

illuminate at least one light source located on the intelligent assistant device to non-verbally communicate that the intelligent assistant device is tracking the location of the first human and a location of the second human.

20. An intelligent assistant device configured to respond to natural language inputs, comprising:

an enclosure;

a plurality of LEDs located around at least a portion of the enclosure;

a plurality of sensors comprising at least one camera and at least one microphone;

at least one speaker;

a logic machine; and

a storage machine holding instructions executable by the logic machine to;

from the at least one camera, receive image data indicating presence of a human;

in response to receiving the image data, illuminate at least one LED of the plurality of LEDs to non-verbally communicate detection of the presence of the human;

from one or more sensors of the plurality of sensors, receive data indicating context information of the human;

using at least the data indicating context information of the human, determine one or more contexts of the human, wherein the one or more contexts of the human comprise (1) an initial identity of the human that corresponds to a previously-identified human and represents an initial confidence value, and/or (2) a verified identity of the human that represents a verified confidence value greater than the initial confidence value; and

in response to determining the one or more contexts of the human, illuminate at least one LED of the plurality of LEDs to non-verbally communicate the one or more contexts of the human.