Non-Line-of-Sight Radar-Based Gesture Recognition

• US 20170192523A1
• Filed: 03/20/2017
• Published: 07/06/2017
• Est. Priority Date: 10/02/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Poupyrev, Ivan
• Granted Patent: US 10,664,059 B2
• Time in Patent Office: Days
• CPC Class Codes: G01S 13/02 Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
• CPC Class Codes: G01S 13/0209 Systems with very large relative bandwidth, i.e. larger than 10 %, e.g. baseband, pulse, carrier-free, ultrawideband
• CPC Class Codes: G01S 13/04 Systems determining presence of a target based on relative movement of target G01S13/56
• CPC Class Codes: G01S 13/72 for two-dimensional tracking, e.g. combination of angle and range tracking, track-while-scan radar
• CPC Class Codes: G01S 13/723 by using numerical data
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 2013/462 using multipath signals
• CPC Class Codes: G01S 2013/464 using only the non-line-of-sight signal(s), e.g. to enable survey of scene 'behind' the target only the indirect signal is evaluated
• CPC Class Codes: G01S 7/40 Means for monitoring or calibrating
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

This document describes techniques and devices for non-line-of-sight radar-based gesture recognition. Through use of the techniques and devices described herein, users may control their devices through in-the-air gestures, even when those gestures are not within line-of-sight of their device'"'"'s sensors. Thus, the techniques enable users to control their devices in many situations in which control is desired but conventional techniques do permit effective control, such as to turn the temperature down in a room when the user is obscured from a thermostat'"'"'s gesture sensor, turn up the volume on a media player when the user is in a different room than the media player, or pause a television program when the user'"'"'s gesture is obscured by a chair, couch, or other obstruction.

Claims

1. A smart device comprising:

a non-line-of-sight gesture recognition system configured to;

provide a radar field;

sense reflections from an interaction that is within the radar field;

recognize, at the smart device associated with the radar field and based on the reflections from the interaction that is within the radar field, a gesture; and

determine that the gesture is associated with a non-line-of-sight smart device; and

a transceiver or network interface configured to pass the gesture from the smart device to the non-line-of-sight smart device in a format usable by the non-line-of-site smart device to recognize the gesture and effective to enable the gesture to control the non-line-of-sight smart device or establish communication with the non-line-of-sight smart device.

2. The smart device of claim 1, wherein the non-line-of-sight gesture recognition system is further configured to provide the radar field having a direct radar field and a reflected radar field, the reflected radar field provided in one or more volumes in the room to which direct line-of-sight from the non-line-of-sight gesture-recognition system is not available, and wherein the interaction is within the reflected radar field, and the sensation of the reflections is from the interaction within the reflected radar field.

3. The smart device of claim 2, wherein the configuration of the non-line-of-sight recognition system to sense the reflections from the interaction within the reflection radar field are received at an antenna of the non-line-of-sight gesture-recognition system and reflected from a surface of the room.

4. The smart device of claim 1, wherein the non-line-of-sight gesture recognition system is further configured to provide the radar field having a direct radar field and a penetration radar field, the penetration radar field provided in one or more volumes in the room to which direct line-of-sight from the non-line-of-sight gesture-recognition system is not available, and wherein the interaction is within the penetration radar field, and the sensation of the reflections is from the interaction within the penetration radar field.

5. The smart device of claim 4, wherein the configuration of the non-line-of-sight recognition system to sense the reflections from the interaction within the penetration radar field are received at an antenna of the non-line-of-sight gesture-recognition system and through an object occluding the interaction within the penetration radar field.

6. The smart device of claim 1, wherein the non-line-of-sight recognition system is further configured to receive a second gesture from a second non-line-of-sight gesture-recognition system, the second non-line-of-sight gesture-recognition system associated with the non-line-of-sight smart device, the second gesture associated with the smart device, and wherein the non-line-of-sight gesture recognition system is further configured to control the smart device based on the second gesture.

7. The smart device of claim 1, wherein the determination that the gesture is associated with the non-line-of-sight smart device maps the gesture to a pre-configured control gesture associated with a control input for an application associated with the non-line-of-sight smart device.

8. The smart device of claim 1, wherein the determination that the gesture is associated with the non-line-of-sight smart device maps the gesture to a pre-configured communication-establishment gesture associated the non-line-of-sight smart device.

9. The smart device of claim 1, wherein the transceiver or the network interface is further configured to establish communication with the non-line-of-sight smart device responsive to determination that the gesture is associated with establishing communication, the establishing of the communication effective to enable control through a future gesture determined at the smart device and sensed through the non-line-of-sight gesture-recognition system of the smart device.

10. The smart device of claim 1, wherein the determination that the gesture is associated with the non-line-of-sight smart device maps the gesture to a unique gesture of a set of unique gestures unique to each of a set of smart devices.

11. A system comprising:

a non-line-of-sight gesture recognition system configured to;

provide a radar field within a room, the radar field including reflected radar fields, the reflected radar fields being reflected off of one or more surfaces of the room and provided in one or more volumes in the room to which direct line-of-sight from the non-line-of-sight gesture-recognition system is not available; and

receive a reflection of a gesture interaction made within the one or more volumes in the room to which direct line-of-sight from the non-line-of-sight gesture-recognition system is not available, the reflection enabling capture of data about the gesture interaction,

one or more processors; and

one or more computer readable storage media having instructions stored thereon that, responsive to execution by the one or more processors, implement a system manager, the system manager configured to;

determine a gesture based on the captured data associated with the reflection of the gesture interaction; and

provide the gesture to a device or application effective to control the device or application.

12. The system of claim 11, wherein the reflection of the gesture interaction is made within one of the reflected radar fields and the reflection is received by an antenna of the non-line-of-sight gesture recognition system from a second reflection off of one of the one or more surfaces of the room.

13. The system of claim 11, wherein providing the radar field within the room further provides penetration radar fields within portions of the one or more volumes of the room in which the direct line-of-sight from the non-line-of-sight gesture-recognition system is not available.

14. The system of claim 13, wherein the reflection of the gesture interaction is made within one of the penetration radar fields.

15. A non-line-of-sight gesture recognition system configured to:

receive or capture an image of a room;

receive or determine dimensions of the room;

determine, based at least in part on the image and the dimensions of the room, a volume of the room in which there is no line-of-sight to the non-line-of-sight gesture-recognition system; and

determine, by providing radar transmissions into the room, a set of transmission paths sufficient to cause a reflected radar field to be provided within at least a portion of the volume, the reflected radar field enabling capture of gesture interactions with the reflected radar field.

16. The non-line-of-sight gesture recognition system of claim 15, wherein the determination of the volume of the room includes to track, with a radar transmission, a person or animal'"'"'s movement in the room and, during a portion of the movement, not receive a reflection from the person or animal or a part of the person or the animal.

17. The non-line-of-sight gesture recognition system of claim 15, wherein the determination of the volume of the room further includes to:

provide a radar field effective to receive reflections from objects in the room; and

determine the volume of the room in which there is no line-of-sight to the non-line-of-sight gesture-recognition system as behind the objects relative to the non-line-of-sight gesture-recognition system.

18. The non-line-of-sight gesture recognition system of claim 15, wherein the determination of the set of transmission paths iteratively provides the radar transmissions and correlating reflections from the iteratively provided radar transmissions.

19. The non-line-of-sight gesture recognition system of claim 15 further configured to:

provide the reflected radar field within the volume;

receive a reflection of a gesture interaction made within the volume;

determine, based on the reflection of the gesture interaction, a gesture made within the volume; and

cause a device or application to be controlled based on the determined gesture.

20. The non-line-of-sight gesture recognition system of claim 15 further configured to determine, based on the dimensions of the room and the image of the room, non-imaged but existing floor, wall, or ceiling areas and wherein the determination of the volume of the room in which there is no line-of-sight to the non-line-of-sight gesture-recognition system is based in part on the non-imaged but existing floor, wall, or ceiling areas.