Advanced Gaming and Virtual Reality Control Using Radar

• US 20190232156A1
• Filed: 04/10/2019
• Published: 08/01/2019
• Est. Priority Date: 10/06/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Amihood, Patrick M., Poupyrev, Ivan
• Granted Patent: US 10,908,696 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G01S 13/56 for presence detection presence detection using near field arrangements G01V3/00, e.g. G01V3/08, G01V3/12; burglar, theft or intruder alarms with electrical actuation G08B13/22 - G08B13/26
• CPC Class Codes: G01S 13/66 Radar-tracking systems; Analogous systems
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 13/90 using synthetic aperture techniques , e.g. synthetic aperture radar [SAR] techniques
• CPC Class Codes: G01S 13/904 SAR modes
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4004 of parts of a radar system
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/412 based on a comparison between measured values and known or stored values
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/25 : Fusion techniques
• CPC Class Codes: G06F 18/253 : of extracted features
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 2203/0384 : Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 2221/2105 : Dual mode as a secondary aspect
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/165 : Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/806 : of extracted features
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 17/02 : using a radio link
• CPC Class Codes: G08C 2201/93 : Remote control using other portable devices, e.g. mobile phone, PDA, laptop
• CPC Class Codes: H04Q 2209/883 : where the sensing device enters an active or inactive mode
• CPC Class Codes: H04Q 9/00 : Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
• CPC Class Codes: H04W 16/28 : using beam steering
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Techniques are described herein that enable advanced gaming and virtual reality control using radar. These techniques enable small motions and displacements to be tracked, even in the millimeter or submillimeter scale, for user control actions even when those actions are optically occluded or obscured.

Claims

1. A computer-implemented method for determining a user control action in a gaming or virtual reality environment, the computer-implemented method comprising:

providing a radar field by a radar system;

receiving a radar signal representing a superposition of reflections of first and second points within the radar field, at least one of the first or second points being on a portion of a user;

determining, based on the radar signal, a relative displacement between the first and second points by spatially resolving the first and second points;

determining the user control action for a gaming or virtual reality application based on the relative displacement between the first and second points;

or

providing the relative displacement between the first and second points effective to enable determination of the user control action by the gaming or virtual reality application;

determining locations in three dimensions, or relative to the user, of the first and second points; and

representing, in a game user interface or a virtual reality world associated with the gaming or virtual reality application, the first and second points;

or

passing the locations of the first and second points to the gaming or virtual reality application effective to enable the gaming or virtual reality application to represent the first and second points within the game user interface or the virtual reality world.

2. The computer-implemented method of claim 1, wherein the providing the relative displacement and the passing the locations of the first and second points are performed through an application programming interface.

3. The computer-implemented method of claim 1, further comprising distinguishing the first and second points within the radar signal based on micro-Doppler centroids for the first and second points.

4. The computer-implemented method of claim 1, wherein spatially resolving the first and second points comprises measuring:

a radial distance for the first and second points using a time delay between transmittal of the radar field and reception of the radar signal;

a radial velocity measured through an observed Doppler frequency for the first and second points; and

a reflected energy for the first and second points.

5. The computer-implemented method of claim 1, wherein the representing, in the game user interface or the virtual reality world associated with the gaming or virtual reality application comprises producing a representation of the portion of the user within the game user interface or the virtual reality world associated with the gaming or virtual reality application.

6. The computer-implemented method of claim 1, wherein the determining the locations of the first and second points is based on a determined position of another portion of the user relative to the first and second points.

7. The computer-implemented method of claim 6, wherein:

the radar signal further represents a superposition of a third point; and

the determined position of the other portion of the user is based on a displacement of the third point relative to the first and second points.

8. The computer-implemented method of claim 1, wherein another of the first or second points is a portion of a gaming implement.

9. The computer-implemented method of claim 8, wherein the gaming implement is a mock game controller, the mock game controller not having control or communication capabilities.

10. The computer-implemented method of claim 8, wherein representing the first and second points within the game user interface or the virtual reality world comprises producing a representation of the gaming implement within the game user interface or the virtual reality world.

11. An apparatus comprising:

one or more computer processors;

a radar system comprising;

one or more radar-emitting elements configured to provide a radar field; and

one or more antenna elements configured to receive a radar signal representing a superposition of reflections of two or more points within the radar field provided by the radar system; and

one or more computer-readable storage media having instructions stored thereon that, responsive to execution by the one or more computer processors, implement a micro-motion tracking module and a control recognition module;

the micro-motion tracking module configured to determine a relative displacement between a first and second point of the two or more points, at least one of the two or more points being on a portion of a user, the determination performed by spatially resolving the first and second points; and

the control recognition module configured to;

determine a user-control action based on the relative displacement between the first and second points; and

pass the user-control action to a game application or virtual reality program.

12. The apparatus of claim 11, wherein the apparatus is configured as smart glasses, a smart watch, a virtual or augmented reality system, a gaming system, a mobile phone, a tablet, a laptop, or a desktop computer.

13. The apparatus of claim 11, wherein the user-control action and the location are passed to the game application or the virtual reality program via an application programming interface.

14. The apparatus of claim 11, wherein the instructions, responsive to execution by the one or more computer processors, further implements a user representation module configured to:

determine a location in three dimensions, or relative to the user, for the first and second points based on the relative displacement between the first and second points and data indicating a position of the user; and

pass the location in three dimension or relative to the user effective to enable the game application or the virtual reality program to represent, in a game user interface or a virtual reality world, respectively, of movement of the first and second points.

15. The apparatus of claim 14, wherein the representation of movement of the first and second points within the game user interface or the virtual reality world comprises producing a representation of the portion of the user within the game user interface or the virtual reality world.

16. The apparatus of claim 14, wherein:

another of the first or second points is a portion of a gaming implement; and

the representation of movement of the first and second points within the game user interface or the virtual reality world comprises producing a representation of the gaming implement within the game user interface or the virtual reality world.

17. The apparatus of claim 14, wherein the micro-motion tracking module, control recognition module, or user representation module is further configured to determine the position of the user.

18. The apparatus of claim 17, wherein the determination of the position of the user is based on a determined displacement of another point of the two or more points of radar field.

19. One or more computer-readable storage media having instructions stored thereon that, responsive to execution by one or more computer processors, cause the processors to:

receive a radar signal representing a superposition of reflections of two or more points within a radar field provided by a radar system;

spatially resolve a first point and a second point of the two or more points, at least one of the first or second points being on a portion of a user;

determine a relative displacement between the first and second points based on the resolving;

determine a user control action for a gaming or virtual reality application based on the relative displacement between the first and second points;

or

provide the relative displacement between the first and second points effective to enable determination of the user control action by the gaming or virtual reality application;

determine locations in three dimensions, or relative to the user, of the first and second point based on the resolving; and

represent, in a game user interface or a virtual reality world associated with the gaming or virtual reality application, the first and second points;

or

pass the locations of the first and second points to the gaming or virtual reality application effective to enable the gaming or virtual reality application to represent the first and second points within the game user interface or the virtual reality world.

20. The one or more computer-readable storage media of claim 19, wherein the locations of the first and second points are used to determine a position or orientation of the portion of the user such that the portion of the user can be represented within the game application or virtual reality program.