Radar-based gesture enhancement for voice interfaces

• US 10,890,653 B2
• Filed: 08/22/2018
• Issued: 01/12/2021
• Est. Priority Date: 08/22/2018
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Giusti, Leonardo, Poupyrev, Ivan, Barbello, Brandon, Amihood, Patrick M.
• Primary Examiner(s): Diallo, Mamadou L
• Application Number:
  US16/108,815
• Publication Number:
  US 20200064458A1
• Time in Patent Office: 874 Days
• CPC Class Codes: G01S 13/426 Scanning radar, e.g. 3D radar G01S13/66 takes precedence
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 13/89 for mapping or imaging
• CPC Class Codes: G01S 7/02 of systems according to group G01S13/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

This document describes techniques and systems that enable radar-based gesture enhancement for voice interfaces. The techniques and systems use a radar field to accurately determine three-dimensional (3D) gestures that can be used instead of, or in combination with, a voice interface to enhance interactions with voice-controllable electronic devices. These techniques allow the user to make 3D gestures from a distance to provide a voice input trigger (e.g., a “listen” gesture), interrupt and correct inaccurate actions by the voice interface, and make natural and precise adjustments to functions controlled by voice commands.

Claims

1. A smartphone, comprising:

a microphone;

a radar system, implemented at least partially in hardware, configured to;

provide a radar field;

sense reflections from an object in the radar field;

analyze the reflections from the object in the radar field; and

provide, based on the analysis of the reflections, radar data;

one or more computer processors; and

one or more computer-readable media having instructions stored thereon that, responsive to execution by the one or more computer processors, implement a radar-based application configured to;

maintain the microphone in a non-operational mode;

detect, based on the radar data, a gesture by the object in the radar field;

determine, based on the radar data, that the gesture is a voice input trigger; and

responsive to determining the gesture is the voice input trigger, cause the microphone to enter an operational mode.

2. The smartphone of claim 1, wherein the non-operational mode is a mode in which:

the microphone is powered off;

or

the microphone is powered on, and the microphone cannot be used to record or act on an audio input.

3. The smartphone of claim 1, wherein:

the object in the radar field is a user; and

the voice input trigger is a three-dimensional (3D) gesture by the user.

4. The smartphone of claim 1, wherein the radar-based application is further configured to cause, responsive to not receiving a voice input within a threshold time of receiving the voice input trigger, the microphone to enter the non-operational mode.

5. The smartphone of claim 1, wherein:

the smartphone further comprises a voice interface module that is configured, responsive to the microphone entering the operational mode, to receive, at a first time, a voice command, the voice command being a command directed to a device that can be interacted with via the radar-based application; and

the radar-based application is further configured to receive, at a second time that is later than the first time and within a threshold time of the voice interface module receiving the voice command, a 3D gesture that specifies a particular device to which the voice command is directed.

6. The smartphone of claim 1, wherein:

the smartphone further comprises a voice interface module that is configured, responsive to the microphone entering the operational mode, to receive, at a first time, a voice command, the voice command being a command to adjust a function of a device that can be interacted with via the radar-based application; and

the radar-based application is further configured to receive, at a second time that is later than the first time and within a threshold time of the voice interface module receiving the voice command, a 3D gesture that specifies an amount of adjustment to the function of the device.

7. The smartphone of claim 1, wherein the radar-based application is further configured, responsive to the microphone entering the operational mode, to:

provide an audio message;

receive, within a duration of the audio message, another 3D gesture, the other 3D gesture corresponding to a command to cease providing the audio message; and

responsive to receiving the other 3D gesture, cease providing the audio message.

8. The smartphone of claim 1, wherein the radar system further comprises a digital beamformer and an angle estimator, and the radar system is configured to monitor angles in a field of view between approximately −

90 degrees and approximately 90 degrees.

9. A system, comprising:

an electronic device that includes a microphone;

a radar system, implemented at least partially in hardware, configured to;

provide a radar field;

sense reflections from an object in the radar field;

analyze the reflections from the object in the radar field; and

provide, based on the analysis of the reflections, radar data;

one or more computer processors; and

one or more computer-readable media having instructions stored thereon that, responsive to execution by the one or more computer processors, implement an interaction manager configured to;

generate a virtual map of an environment, the virtual map identifying a location and a type of one or more devices in the environment that can be interacted with via the interaction manager;

receive, at a first time, a voice command directed to at least one of the one or more identified devices, the voice command including at least the type of the at least one device;

determine, at a second time that is later than, or at approximately a same time as, the first time, a three-dimensional (3D) gesture, the determination based on the radar data, the 3D gesture corresponding to a sub-command, the sub-command related to the voice command; and

responsive to the voice command and the 3D gesture, cause the at least one of the one or more devices to perform an action corresponding to the voice command and the sub-command.

10. The system of claim 9, wherein:

the electronic device includes an image-capture device; and

the interaction manager is further configured to;

generate the virtual map based on a scan of the environment by the image-capture device; and

responsive to a determination that the image-capture device has stopped moving, cause the image-capture device to enter a non-operational mode.

11. The system of claim 9, wherein the interaction manager is further configured to generate the virtual map by:

receiving, from the one or more devices that can be interacted with by the radar-based application, an identification signal, the identification signal including the location and the type of the one or more devices;

or

receiving, via input from a user, identification of one or more devices that can be interacted with by the radar-based application, the identification including the location and the type of the one or more devices.

12. The system of claim 9, wherein the sub-command corresponding to the 3D gesture specifies, based on the identified location, a particular one of the at least one of the one or more devices.

13. The system of claim 9, wherein:

the voice command is a command to adjust a function of the at least one of the one or more identified devices; and

the sub-command corresponding to the 3D gesture specifies an amount of adjustment to the function of the at least one of the identified devices.

14. The system of claim 9, wherein the sub-command is a command that adds to, restricts, directs, or fine-tunes the voice command.

15. A method implemented in an electronic device that includes a radar system, a radar-based application, and a microphone, the method comprising:

providing, by the radar system, a radar field;

sensing, by the radar system, reflections from an object in the radar field;

analyzing the reflections from the object in the radar field;

providing, based on the analysis of the reflections, radar data;

maintaining, by the radar-based application, the microphone in a non-operational mode;

detecting, based on the radar data, a gesture by the object in the radar field;

determining, based on the radar data, that the gesture is a voice input trigger; and

responsive to determining the gesture is the voice input trigger, causing the microphone to enter an operational mode.

16. The method of claim 15, wherein the non-operational mode is a mode in which:

the microphone is powered off;

or

the microphone is powered on, and the microphone cannot be used to record or act on an audio input.

17. The method of claim 15, further comprising, responsive to not receiving a voice input within a threshold time of receiving the voice input trigger, causing the microphone to enter the non-operational mode.

18. The method of claim 15, wherein the object in the radar field is a user and the gesture is:

a three-dimensional (3D) gesture by the user;

or

a movement by the user to within a threshold distance from the smartphone.

19. The method of claim 15, wherein the electronic device further comprises a voice interface module, and the method further comprises, responsive to the microphone entering the operational mode:

receiving, at a first time and by the voice interface module, a voice command, the voice command being a command directed to a device that can be interacted with via the radar-based application; and

receiving, by the radar-based application, at a second time that is later than the first time and within a threshold time of the voice interface module receiving the voice command, a 3D gesture that specifies a particular device to which the voice command is directed.

20. The method of claim 15, wherein the electronic device further comprises a voice interface module, and the method further comprises, responsive to the microphone entering the operational mode:

receiving, at a first time and by the voice interface module, a voice command, the voice command being a command to adjust a function of a device that can be interacted with via the radar-based application; and

receiving, by the radar-based application, at a second time that is later than the first time and within a threshold time of the voice interface module receiving the voice command, a 3D gesture that specifies an amount of adjustment to the function of the device.