Radar-based contextual sensing

• US 10,222,469 B1
• Filed: 10/06/2016
• Issued: 03/05/2019
• Est. Priority Date: 10/06/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Gillian, Nicholas Edward, Schwesig, Carsten C., Lien, Jaime, Amihood, Patrick M., Poupyrev, Ivan
• Primary Examiner(s): Akonai, Olumide Ajibade
• Application Number:
  US15/287,200
• Time in Patent Office: 880 Days
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G01S 13/56 for presence detection presence detection using near field arrangements G01V3/00, e.g. G01V3/08, G01V3/12; burglar, theft or intruder alarms with electrical actuation G08B13/22 - G08B13/26
• CPC Class Codes: G01S 13/66 Radar-tracking systems; Analogous systems
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 13/90 using synthetic aperture techniques , e.g. synthetic aperture radar [SAR] techniques
• CPC Class Codes: G01S 13/904 SAR modes
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4004 of parts of a radar system
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/412 based on a comparison between measured values and known or stored values
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/25 : Fusion techniques
• CPC Class Codes: G06F 18/253 : of extracted features
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 2203/0384 : Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 2221/2105 : Dual mode as a secondary aspect
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/165 : Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/806 : of extracted features
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 17/02 : using a radio link
• CPC Class Codes: G08C 2201/93 : Remote control using other portable devices, e.g. mobile phone, PDA, laptop
• CPC Class Codes: H04Q 2209/883 : where the sensing device enters an active or inactive mode
• CPC Class Codes: H04Q 9/00 : Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
• CPC Class Codes: H04W 16/28 : using beam steering
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

This document describes apparatuses and techniques for radar-based contextual sensing. In some aspects, a radar sensor of a device is activated to obtain radar data for a space of interest. Three-dimensional (3D) radar features are extracted from the radar data and positional data is received from sensors. Based on the positional data, spatial relation of the 3D radar features is determined to generate a set of 3D landmarks for the space. This set of 3D landmarks is then compared with known 3D context models to identify a 3D context model that matches the 3D landmarks. Based on a matching 3D context model, a context for the space is retrieved and used to configure contextual settings of the device. By so doing, contextual settings of the device be dynamically configured to address changes in context or for different device environments.

Claims

1. A computer-implemented method comprising:

activating a radar sensor of a device to obtain radar data for a space of interest;

extracting, from the radar data, three-dimensional (3D) radar features of the space of interest;

receiving, from a sensor of the device, positional data, the sensor comprising an accelerometer, a gyroscope, a hall effect sensor, a magnetometer, a temperature sensor, an acoustic sensor, a capacitive sensor, a proximity sensor, an ambient light sensor, a red-green-blue (RGB) sensor, an infrared sensor, or a depth sensor;

determining, based on the positional data, a spatial relation of the 3D radar features;

generating, based on the spatial relation and the 3D radar features, a set of 3D landmarks of the space of interest;

comparing the set of 3D landmarks to known 3D context models;

retrieving, based on a matching 3D context model, a context associated with the space of interest; and

configuring context settings of the device based on the context associated with the space of interest.

2. The computer-implemented method of claim 1, wherein the 3D radar features comprise one or more of a radar reflection feature, a radar detection feature, a radar position feature, or a radar shape feature.

3. The computer-implemented method of claim 2, wherein the 3D radar features indicate a number of targets, total reflected energy, two-dimensional spatial coordinates, three-dimensional spatial coordinates, or a one-dimensional spatial dispersion.

4. The computer-implemented method of claim 1, wherein determining the spatial relation of the 3D radar features includes using the positional data to implement a synthetic aperture through which the 3D radar features are captured or processed.

5. The computer-implemented method of claim 1, wherein the context associated with the space of interest indicates a level of privacy associated with the space, a level of security associated with the space, a type of activities associated with the space, or functionalities provided by other devices in the space.

6. The computer-implemented method of claim 1, wherein the context settings of the device comprise a ringer volume setting, audio playback setting, ringer mode setting, display mode setting, network connectivity setting, remote control functionalities, security setting, or privacy setting.

7. The computer-implemented method of claim 1, wherein the sensor is the accelerometer or the gyroscope and the positional data represents motion of the device.

8. The computer-implemented method of claim 1, wherein the sensor is the hall effect sensor or the magnetometer and the positional data represents a magnetic field proximate the device.

9. The computer-implemented method of claim 1, wherein the sensor is the temperature sensor and the positional data represents a temperature proximate the device.

10. The computer-implemented method of claim 1, wherein the sensor is the acoustic sensor and the positional data represents sound proximate the device.

11. The computer-implemented method of claim 1, wherein the sensor is the capacitive sensor and the positional data represents a touch input, proximity with a user or object, or a property of a material proximate the device.

12. The computer-implemented method of claim 1, wherein the sensor is the proximity sensor and the positional data represents a presence of a user or object proximate the device.

13. The computer-implemented method of claim 1, wherein the sensor is the ambient light sensor and the positional data represents an amount of light proximate the device.

14. The computer-implemented method of claim 1, wherein the sensor is the RGB sensor and the positional data represents a visual representation of a user or an object proximate the device.

15. The computer-implemented method of claim 1, wherein the sensor is the infrared sensor and the positional data represents thermal information about a user or an object proximate the device.

16. The computer-implemented method of claim 1, wherein the sensor is the depth sensor and the positional data represents a distance to a user or an object proximate the device.

17. An apparatus comprising:

one or more computer processors;

a radar sensor comprising;

a radar-emitting element configured to provide a radar field;

a radar-receiving element configured to receive one or more reflection signals that correspond to the radar field;

a sensor configured to sense position-related aspects of the apparatus, the sensor comprising an accelerometer, a gyroscope, a hall effect sensor, a magnetometer, a temperature sensor, an acoustic sensor, a capacitive sensor, a proximity sensor, an ambient light sensor, a red-green-blue (RGB) sensor, an infrared sensor, or a depth sensor;

one or more computer-readable storage media having instructions stored thereon that, responsive to execution by the one or more computer processors, implement a context manager to;

activate the radar sensor of a device to obtain radar data for a space in which the apparatus operates;

extract, from the radar data, three-dimensional (3D) radar features of the space;

receive, from the sensor, positional data indicating the position-related aspects of the apparatus;

determine, based on the positional data, a spatial relation of the 3D radar features;

generate, based on the spatial relation and the 3D radar features, a set of 3D landmarks of the space;

compare the set of 3D landmarks to known 3D context models;

retrieve, based on a matching 3D context model, a context associated with the space; and

configure context settings of the apparatus based on the context associated with the space in which the apparatus operates.

18. The apparatus of claim 17, wherein the 3D radar features comprise one or more of a radar reflection feature, a radar detection feature, a radar position feature, or a radar shape feature.

19. The apparatus of claim 17, wherein determining the spatial relation of the 3D radar features includes using the positional data to implement a synthetic aperture through which the 3D radar features are captured or processed.

20. The apparatus of claim 17, wherein the radar sensor further comprises an antenna array through which the radar-emitting element provides the radar field and the radar-receiving element receives the one or more reflection signals.

21. The apparatus of claim 17, wherein the context associated with the space of interest indicates a level of privacy associated with the space, a level of security associated with the space, a type of activities associated with the space, or functionalities provided by devices in the space.

22. The apparatus of claim 17, wherein the apparatus is embodied as a smart-phone, smart-glasses, smart-watch, tablet computer, laptop computer, set-top box, smart-appliance, home automation controller, or television.

23. The apparatus of claim 17, wherein the sensor is the accelerometer or the gyroscope and the position-related aspects of the apparatus represent motion of the apparatus.

24. The apparatus of claim 17, wherein the sensor is the hall effect sensor or the magnetometer and the position-related aspects of the apparatus represent a magnetic field proximate the apparatus.

25. The apparatus of claim 17, wherein the sensor is the temperature sensor and the position-related aspects of the apparatus represent a temperature proximate the apparatus.

26. The apparatus of claim 17, wherein the sensor is the acoustic sensor and the position-related aspects of the apparatus represent sound proximate the apparatus.

27. The apparatus of claim 17, wherein the sensor is the capacitive sensor and the position-related aspects of the apparatus represent a touch input, proximity with a user or object, or a property of a material proximate the apparatus.

28. The apparatus of claim 17, wherein the sensor is the proximity sensor and the position-related aspects of the apparatus represent a presence of a user or object proximate the apparatus.

29. The apparatus of claim 17, wherein the sensor is the ambient light sensor and the position-related aspects of the apparatus represent an amount of light proximate the apparatus.

30. The apparatus of claim 17, wherein the sensor is the RGB sensor and the position-related aspects of the apparatus represent a visual representation of a user or an object proximate the apparatus.

31. The apparatus of claim 17, wherein the sensor is the infrared sensor the position-related aspects of the apparatus represent thermal information about a user or an object proximate the apparatus.

32. The apparatus of claim 17, wherein the sensor is the depth sensor the position-related aspects of the apparatus represent a distance to a user or an object proximate the apparatus.

33. A system-on-chip comprising:

a radar-emitting element configured to provide a radar field;

a radar-receiving element configured to receive reflection signals;

an antenna array through which the radar field is provided and through which the reflection signals are received;

a data interface configured to enable communication with one or more sensors;

a processor; and

a computer-readable storage media having instructions stored thereon that, responsive to execution by the computer processor, implement a context manager to;

activate the radar sensor to obtain radar data for a space of interest;

extract, from the radar data, three-dimensional (3D) radar features of the space of interest;

receive, from the data interface, positional data from a sensor, the sensor comprising an accelerometer, a gyroscope, a hall effect sensor, a magnetometer, a temperature sensor, an acoustic sensor, a capacitive sensor, a proximity sensor, an ambient light sensor, a red-green-blue (RGB) sensor, an infrared sensor, or a depth sensor;

determine, based on the positional data, a spatial relation of the 3D radar features;

generate, based on the spatial relation and the 3D radar features, a set of 3D landmarks of the space of interest;

compare the set of 3D landmarks to known 3D context models;

retrieve, based on a matching 3D context model, a context associated with the space of interest; and

configure context settings of a device in which the system-on-chip is implemented based on the context associated with the space of interest.

34. The system-on-chip of claim 33, further comprising a memory storing the known 3D context models or a data interface through which the known 3D context models are accessible.

35. The system-on-chip of claim 33, wherein the 3D radar features comprise one or more of a radar reflection feature, a radar detection feature, a radar position feature, or a radar shape feature.

36. The system-on-chip of claim 33, wherein the context associated with the space of interest indicates a level of privacy associated with the space, a level of security associated with the space, a type of activities associated with the space, or functionalities provided by other devices in the space.

37. The system-on-chip of claim 33, wherein the antenna array comprises at least two on-chip antennas operably connected to the radar-emitting element and at least two on-chip antennas operably connected to the radar-receiving element.

38. The system-on-chip of claim 37, wherein the radar-emitting element configured to provide a radar field, radar-receiving element, and antenna array are further configured to provide reflected signals suitable for transformation into at least four channels of radar data.