Radar-Enabled Sensor Fusion

• US 20190321719A1
• Filed: 07/03/2019
• Published: 10/24/2019
• Est. Priority Date: 10/06/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Gillian, Nicholas Edward, Schwesig, Carsten C., Lien, Jaime, Amihood, Patrick M., Poupyrev, Ivan
• Granted Patent: US 11,132,065 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G01S 13/56 for presence detection presence detection using near field arrangements G01V3/00, e.g. G01V3/08, G01V3/12; burglar, theft or intruder alarms with electrical actuation G08B13/22 - G08B13/26
• CPC Class Codes: G01S 13/66 Radar-tracking systems; Analogous systems
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 13/90 using synthetic aperture techniques , e.g. synthetic aperture radar [SAR] techniques
• CPC Class Codes: G01S 13/904 SAR modes
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4004 of parts of a radar system
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/412 based on a comparison between measured values and known or stored values
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/25 : Fusion techniques
• CPC Class Codes: G06F 18/253 : of extracted features
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 2203/0384 : Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 2221/2105 : Dual mode as a secondary aspect
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/165 : Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/806 : of extracted features
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 17/02 : using a radio link
• CPC Class Codes: G08C 2201/93 : Remote control using other portable devices, e.g. mobile phone, PDA, laptop
• CPC Class Codes: H04Q 2209/883 : where the sensing device enters an active or inactive mode
• CPC Class Codes: H04Q 9/00 : Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
• CPC Class Codes: H04W 16/28 : using beam steering
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

This document describes apparatuses and techniques for radar-enabled sensor fusion. In some aspects, a radar field is provided and reflection signals that correspond to a target in the radar field are received. The reflection signals are transformed to provide radar data, from which a radar feature indicating a physical characteristic of the target is extracted. Based on the radar features, a sensor is activated to provide supplemental sensor data associated with the physical characteristic. The radar feature is then augmented with the supplemental sensor data to enhance the radar feature, such as by increasing an accuracy or resolution of the radar feature. By so doing, performance of sensor-based applications, which rely on the enhanced radar features, can be improved.

Claims

1. An apparatus comprising:

at least one computer processor;

at least one radar sensor comprising;

at least one radar-emitting element configured to provide radar fields; and

at least one radar-receiving element configured to receive radar reflection signals caused by reflections of the radar fields off objects within the radar fields;

a plurality of sensors, each of the sensors comprising a distinct type of sensor configured to sense a respective environmental variance of the apparatus; and

at least one computer-readable storage medium having instructions stored thereon that, responsive to execution by the computer processor, cause the apparatus to;

provide, via the radar-emitting element, a radar field;

receive, via the radar-receiving element, a reflection signal caused by a reflection of the radar field off an object within the radar field;

determine, based on the received reflection signal, at least one of a radar detection feature, a radar reflection feature, a radar motion feature, a radar position feature, or a radar shape feature of the object;

select, based on the determined radar detection, motion, position, or shape feature of the object, one of the sensors to provide supplemental sensor data;

receive the supplemental sensor data from the selected sensor;

augment the radar detection, motion, position, or shape feature of the object with the supplemental sensor data to enhance the radar detection, motion, position, or shape feature of the object; and

provide the enhanced radar detection, motion, position, or shape feature of the object to a sensor-based application effective to improve performance of the sensor-based application.

2. The apparatus of claim 1, wherein:

the instructions further cause the apparatus to determine an indication of a number of targets, total reflected energy, moving energy, one-dimensional velocity, one-dimensional velocity dispersion, three-dimensional spatial coordinates, or one-dimensional spatial dispersion based on the reflection signal; and

the selecting is further based on the indication.

3. The apparatus of claim 1, wherein:

the instructions further cause the apparatus to perform a range-Doppler transform, range profile transform, micro-Doppler transform, I/Q transform, or spectrogram transform on the reflection signal; and

the determining is further based on the performed transform.

4. The apparatus of claim 1, wherein the sensor-based application comprises proximity detection, user detection, user tracking, activity detection, facial recognition, breathing detection, or motion signature identification.

5. The apparatus of claim 1, wherein the sensors comprise two or more of an accelerometer, gyroscope, hall effect sensor, magnetometer, temperature sensor, microphone, capacitive sensor, proximity sensor, ambient light sensor, red-green-blue (RGB) image sensor, infrared sensor, or depth sensor.

6. The apparatus of claim 1, wherein augmenting the radar detection, motion, position, or shape feature of the object with the supplemental sensor data is effective to enhance the radar detection, motion, position, or shape feature of the object by:

increasing a positional accuracy of the radar detection, motion, position, or shape feature of the object;

mitigating a false-positive detection attributed to the radar detection, motion, position, or shape feature of the object;

increasing a spatial resolution of the radar detection, motion, position, or shape feature of the object;

increasing a surface resolution of the radar detection, motion, position, or shape feature of the object;

or

improving a classification precision of the radar detection, motion, position, or shape feature of the object.

7. The apparatus of claim 1, wherein:

the radar sensor consumes less power than the selected sensor; and

the selected sensor is in a low-power state until the selection.

8. The apparatus of claim 1, wherein the apparatus is embodied as a smart-phone, smart-glasses, smart-watch, tablet computer, laptop computer, set-top box, smart-appliance, home automation controller, or television.

9. The apparatus of claim 1, wherein:

the radar-receiving element comprises a plurality of antennas; and

the instructions further cause the apparatus to receive the reflection signal, via the antennas, using beamforming techniques.

10. A method comprising:

providing, via at least one radar-emitting element of a device, a radar field;

receiving, via at least one radar-receiving element of the device, a reflection signal caused by a reflection of the radar field off a target within the radar field;

abstracting, from the reflection signal, a radar feature that indicates a physical characteristic of the target;

selecting, based on the radar feature being a first type of radar feature, a first sensor of a plurality of sensors of the device;

or

selecting, based on the radar feature being a second type of radar feature, a second sensor of the sensors of the device that is a different type of sensor than the first sensor;

receiving supplemental sensor data from the selected sensor, the supplemental sensor data describing an environmental variance corresponding to the distinct type of the selected sensor that further indicates the physical characteristic of the target;

augmenting the radar feature with the supplemental sensor data; and

providing the augmented radar feature to a sensor-based application.

11. The method of claim 10, wherein the first sensor is configured to detect a first environmental variance around the target and the second sensor is configured to detect a second environmental variance around the target.

12. The method of claim 10, wherein the first or second type of the radar feature comprises a surface feature, motion feature, position feature, or detection feature.

13. The method of claim 10, wherein the sensors of the device comprise at least two of an accelerometer, gyroscope, hall effect sensor, magnetometer, temperature sensor, microphone, capacitive sensor, proximity sensor, ambient light sensor, red-green-blue (RGB) image sensor, infrared sensor, or depth sensor.

14. The method of claim 10, wherein the radar feature indicates a number of targets in the radar field, total reflected energy, moving energy, one-dimensional velocity, one-dimensional velocity dispersion, three-dimensional spatial coordinates, or one-dimensional spatial dispersion.

15. At least one computer-readable storage media device comprising instructions that, when executed by a processing system, cause the processing system to:

cause at least one radar-emitting element to provide a radar field;

receive, via at least one radar-receiving element, a reflection signal caused by reflection of the radar field off a target within the radar field;

resolve the reflection signal into at least one of a radar detection feature, a radar reflection feature, a radar motion feature, a radar position feature, or a radar shape feature of the target causing the reflection signal;

select, based on the radar detection, reflection, motion, position, or shape feature of the target, at least one sensor to provide supplemental sensor data for the radar detection, reflection, motion, position, or shape feature of the target, the supplemental sensor data indicating at least one environmental variance around the target;

receive, from the selected sensor, the supplemental sensor data indicating the environmental variance;

augment the radar detection, reflection, motion, position, or shape feature of the target with the supplemental sensor data indicating the environmental variance to provide an enhanced radar detection, reflection, motion, position, or shape feature of the target; and

expose the enhanced radar detection, reflection, motion, position, or shape feature of the target to a sensor-based application effective to improve performance of the sensor-based application.

16. The computer-readable storage media device of claim 15, wherein the selected sensor comprises an accelerometer, gyroscope, hall effect sensor, magnetometer, temperature sensor, microphone, capacitive sensor, proximity sensor, ambient light sensor, red-green-blue (RGB) image sensor, infrared sensor, or depth sensor.

17. The computer-readable storage media device of claim 15, wherein the instructions further cause the processing system to determine an indication of a number of targets, total reflected energy, moving energy, one-dimensional velocity, one-dimensional velocity dispersion, three-dimensional spatial coordinates, or one-dimensional spatial dispersion based on the reflection signal;

and

the selecting is further based on the indication.

18. The computer-readable storage media device of claim 15, wherein:

the instructions further cause the processing system to perform a range-Doppler transform, range profile transform, micro-Doppler transform, I/Q transform, or spectrogram transform on the reflection signal; and

the resolving is further based on the performed transform.

19. The computer-readable storage media device of claim 15, wherein the sensor-based application comprises proximity detection, user detection, user tracking, activity detection, facial recognition, breathing detection, or motion signature identification.

20. The computer-readable storage media device of claim 15, wherein augmenting the radar detection, motion, position, or shape feature of the object with the supplemental sensor data is effective to enhance the radar detection, motion, position, or shape feature of the object by:

increasing a positional accuracy of the radar detection, motion, position, or shape feature of the object;

mitigating a false positive detection attributed to the radar detection, motion, position, or shape feature of the object;

increasing a spatial resolution of the radar detection, motion, position, or shape feature of the object;

increasing a surface resolution of the radar detection, motion, position, or shape feature of the object;

or

improving a classification precision of the radar detection, motion, position, or shape feature of the object.