Radar-Enabled Sensor Fusion

• US 20170097413A1
• Filed: 10/05/2016
• Published: 04/06/2017
• Est. Priority Date: 10/06/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Gillian, Nicholas Edward, Amihood, Patrick M., Schwesig, Carsten C., Lien, Jaime, Poupyrev, Ivan
• Granted Patent: US 10,401,490 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G01S 13/56 for presence detection presence detection using near field arrangements G01V3/00, e.g. G01V3/08, G01V3/12; burglar, theft or intruder alarms with electrical actuation G08B13/22 - G08B13/26
• CPC Class Codes: G01S 13/66 Radar-tracking systems; Analogous systems
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 13/90 using synthetic aperture techniques , e.g. synthetic aperture radar [SAR] techniques
• CPC Class Codes: G01S 13/904 SAR modes
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4004 of parts of a radar system
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/412 based on a comparison between measured values and known or stored values
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/25 : Fusion techniques
• CPC Class Codes: G06F 18/253 : of extracted features
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 2203/0384 : Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 2221/2105 : Dual mode as a secondary aspect
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/165 : Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/806 : of extracted features
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 17/02 : using a radio link
• CPC Class Codes: G08C 2201/93 : Remote control using other portable devices, e.g. mobile phone, PDA, laptop
• CPC Class Codes: H04Q 2209/883 : where the sensing device enters an active or inactive mode
• CPC Class Codes: H04Q 9/00 : Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
• CPC Class Codes: H04W 16/28 : using beam steering
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

This document describes apparatuses and techniques for radar-enabled sensor fusion. In some aspects, a radar field is provided and reflection signals that correspond to a target in the radar field are received. The reflection signals are transformed to provide radar data, from which a radar feature indicating a physical characteristic of the target is extracted. Based on the radar features, a sensor is activated to provide supplemental sensor data associated with the physical characteristic. The radar feature is then augmented with the supplemental sensor data to enhance the radar feature, such as by increasing an accuracy or resolution of the radar feature. By so doing, performance of sensor-based applications, which rely on the enhanced radar features, can be improved.

Claims

1. A computer-implemented method comprising:

providing, via a radar-emitting element of a device, a radar field;

receiving, via a radar-receiving element of the device, one or more reflection signals that correspond to a target in the radar field;

transforming the one or more reflection signals to provide radar data;

extracting, from the radar data, a radar feature that indicates a physical characteristic of the target;

activating, based on the radar feature, a sensor of the device to provide supplemental sensor data associated with the physical characteristic;

augmenting the radar feature with the supplemental sensor data to enhance the radar feature; and

providing the enhanced radar feature to a sensor-based application effective to improve performance of the sensor-based application.

2. The computer-implemented method of claim 1, wherein the radar feature comprises one of target detection, target reflection, target motion, target position, or target shape.

3. The computer-implemented method of claim 2, wherein the radar feature indicates a number of targets, total reflected energy, moving energy, one-dimensional velocity, one-dimensional velocity dispersion, three-dimensional spatial coordinates, or one-dimensional spatial dispersion.

4. The computer-implemented method of claim 1, wherein transforming the one or more reflection signals comprises performing one of a range-Doppler transform, range profile transform, micro-Doppler transform, I/Q transform, or spectrogram transform.

5. The computer-implemented method of claim 1, wherein the sensor-based application comprises proximity detection, user detection, user tracking, activity detection, facial recognition, breathing detection, or motion signature identification.

6. The computer-implemented method of claim 1, wherein the sensor that provides the supplemental sensor data comprises an accelerometer, gyroscope, hall effect sensor, magnetometer, temperature sensor, microphone, capacitive sensor, proximity sensor, ambient light sensor, red-green-blue (RGB) image sensor, infrared sensor, or depth sensor.

7. The computer-implemented method of claim 1, wherein augmenting the radar feature with the supplemental sensor data is effective to enhance the radar feature by:

increasing a positional accuracy of the radar feature;

mitigating a false positive detection attributed to the radar feature;

increasing a spatial resolution of the radar feature;

increasing a surface resolution of the radar feature;

or

improving a classification precision of the radar feature.

8. The computer-implemented method of claim 1, wherein a radar sensor of the device includes the radar-emitting element and radar-receiving element, the radar sensor consumes less power than the sensor that is activated, and the radar sensor is operated while the sensor is in a low-power state until a change in the one or more reflection signals is detected is effective to conserve power of the device.

9. An apparatus comprising:

one or more computer processors;

a radar sensor comprising;

a radar-emitting element configured to provide a radar field;

a radar-receiving element configured to receive one or more reflection signals that correspond to the radar field;

a sensor configured to sense an environmental variance of the apparatus;

one or more computer-readable storage media having instructions stored thereon that, responsive to execution by the one or more computer processors, implement a sensor fusion engine to;

provide, via a radar-emitting element of a device, a radar field;

receive, via a radar-receiving element of the device, one or more reflection signals that correspond to a target in the radar field;

transform the one or more reflection signals to provide radar data;

abstract, from the radar data, a radar feature that indicates a physical characteristic of the target;

activate, based on the radar feature, a sensor of the device to provide supplemental sensor data for the physical characteristic of the target;

augment the radar feature with the supplemental sensor data; and

provide the augmented radar feature to a sensor-based application.

10. The apparatus of claim 9, wherein the apparatus includes multiple sensors to sense multiple respective types of environmental variances, and the sensor fusion engine is further configured to determine a type of the radar feature and select, based on the type of the radar feature, which of the multiple sensors to activate to provide the supplemental sensor data.

11. The apparatus of claim 9, wherein the sensor configured to sense the environmental variance includes an accelerometer, gyroscope, hall effect sensor, magnetometer, temperature sensor, microphone, capacitive sensor, proximity sensor, ambient light sensor, red-green-blue (RGB) image sensor, infrared sensor, or depth sensor.

12. The apparatus of claim 9, wherein the radar feature indicates a number of targets in the radar field, total reflected energy, moving energy, one-dimensional velocity, one-dimensional velocity dispersion, three-dimensional spatial coordinates, or one-dimensional spatial dispersion.

13. The apparatus of claim 9, wherein the radar sensor further comprises an antenna array through which the radar-emitting element provides the radar field and the radar-receiving element receives the one or more reflection signals.

14. The apparatus of claim 13, wherein the antenna array comprises at least two antennas coupled directly to the radar-emitting element and at least two antennas coupled directly to the radar-receiving element.

15. The apparatus of claim 9, wherein the apparatus is embodied as a smart-phone, smart-glasses, smart-watch, tablet computer, laptop computer, set-top box, smart-appliance, home automation controller, or television.

16. A system-on-chip comprising:

a radar-emitting element configured to provide a radar field;

a radar-receiving element configured to receive reflection signals;

an antenna array through which the radar field is provided and through which the reflection signals are received;

a data interface configured to enable communication with one or more sensors;

a processor; and

a computer-readable storage media having instructions stored thereon that, responsive to execution by the computer processor, implement a sensor fusion engine to;

cause the radar-emitting element to provide a radar field;

receive, via the radar-receiving element, one or more of the reflection signals that correspond to the radar field;

transform the one or more reflection signals to provide radar data;

resolve the radar data into a radar feature;

receive, from the one or more sensors, sensor data that indicates an environmental variance;

augment the radar feature with the sensor data that indicates the environmental variance to provide an enhanced radar feature; and

expose the enhanced radar feature to a sensor-based application.

17. The system-on-chip of claim 16, wherein the radar feature comprises one of target detection, target reflection, target motion, target position, or target shape.

18. The system-on-chip of claim 16, wherein the one or more sensors comprise one of an accelerometer, gyroscope, hall effect sensor, magnetometer, temperature sensor, microphone, capacitive sensor, proximity sensor, ambient light sensor, red-green-blue (RGB) image sensor, infrared sensor, or depth sensor.

19. The system-on-chip of claim 16, wherein the antenna array comprises at least two on-chip antennas operably connected to the radar-emitting element and at least two on-chip antennas operably connected to the radar-receiving element.

20. The system-on-chip of claim 19, wherein the radar-emitting element configured to provide a radar field, radar-receiving element, and antenna array are further configured to provide reflected signals suitable for transformation into at least four channels of radar data.