Fine-motion virtual-reality or augmented-reality control using radar

• US 10,540,001 B1
• Filed: 10/06/2016
• Issued: 01/21/2020
• Est. Priority Date: 10/06/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Poupyrev, Ivan, Amihood, Patrick M.
• Primary Examiner(s): Awad, Amr A
• Application Number:
  US15/287,253
• Time in Patent Office: 1,202 Days
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G01S 13/56 for presence detection presence detection using near field arrangements G01V3/00, e.g. G01V3/08, G01V3/12; burglar, theft or intruder alarms with electrical actuation G08B13/22 - G08B13/26
• CPC Class Codes: G01S 13/66 Radar-tracking systems; Analogous systems
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 13/90 using synthetic aperture techniques , e.g. synthetic aperture radar [SAR] techniques
• CPC Class Codes: G01S 13/904 SAR modes
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4004 of parts of a radar system
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/412 based on a comparison between measured values and known or stored values
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/25 : Fusion techniques
• CPC Class Codes: G06F 18/253 : of extracted features
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 2203/0384 : Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 2221/2105 : Dual mode as a secondary aspect
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/165 : Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/806 : of extracted features
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 17/02 : using a radio link
• CPC Class Codes: G08C 2201/93 : Remote control using other portable devices, e.g. mobile phone, PDA, laptop
• CPC Class Codes: H04Q 2209/883 : where the sensing device enters an active or inactive mode
• CPC Class Codes: H04Q 9/00 : Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
• CPC Class Codes: H04W 16/28 : using beam steering
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

This document describes techniques for fine-motion virtual-reality or augmented-reality control using radar. These techniques enable small motions and displacements to be tracked, even in the millimeter or sub-millimeter scale, for user control actions even when those actions are small, fast, or obscured due to darkness or varying light. Further, these techniques enable fine resolution and real-time control, unlike conventional RF-tracking or optical-tracking techniques.

Claims

1. A computer-implemented method for fine-motion virtual-reality or augmented-reality (VR/AR) control, the computer-implemented method comprising:

presenting a VR/AR controller within a virtual world or an augmented-reality viewport;

tracking user actions interacting with the VR/AR controller, the tracked interactions including fine-motions in a millimeter or sub-millimeter range, the tracking comprising;

receiving a radar signal representing a superposition of reflections of two or more points within a radar field provided by a radar-system;

calculating a radial distance for each of the points using a time delay between a transmittal of the radar field and the receiving of the radar signal;

calculating a radial velocity for each of the points using an observed Doppler frequency for each of the points;

ascertaining a reflected energy for each of the points; and

determining a relative displacement, or a relative velocity, between the points based on the radial distance, radial velocity, and reflected energy for each of the points;

altering an orientation of the VR/AR controller in real time and with fine-motion appearance changes, the fine-motion appearance changes in the millimeter or sub-millimeter range and corresponding to the tracked user actions; and

responsive to the altered orientation of the VR/AR controller corresponding to a control orientation, passing a control input corresponding to the control orientation effective to control the virtual world, the augmented-reality world, or an application or device external to the virtual world or the augmented-reality world.

2. The computer-implemented method of claim 1, wherein presenting the VR/AR controller presents the VR/AR controller within the VR world, and further comprising presenting, in the virtual world, a portion of a user performing the user actions interacting with the VR/AR controller.

3. The computer-implemented method of claim 2, wherein presenting the portion of the user presents the portion in real time and in the millimeter or sub-millimeter range.

4. The computer-implemented method of claim 3, wherein presenting the portion of the user shows the portion of the user in contact with the VR/AR controller.

5. The computer-implemented method of claim 1, further comprising distinguishing the points by tracking temporal changes in the radar signal, the distinguishing prior to determining the relative displacement or the relative velocity.

6. The computer-implemented method of claim 1, wherein altering the orientation of the VR/AR controller shows a turn or lateral movement of the VR/AR controller, the turn or the lateral movement shown in real time and corresponding to a real motion in a real world of a velocity and displacement of a portion of a user that performs the user actions.

7. The computer-implemented method of claim 1, wherein the VR/AR controller is presented within the virtual world in three dimensions.

8. The computer-implemented method of claim 1, wherein the VR/AR controller is presented in the augmented-reality viewport, the VR/AR controller presented touching a portion of a user that is performing the user actions.

9. The method of claim 1, wherein the tracked interactions correspond to the relative displacement or relative velocity between the points.

10. The method of claim 1, wherein the points are on one or more hands of a user that performs the user actions.

11. The method of claim 1, wherein a computing device used to perform the method is proximal a head of a user that performs the user actions;

and

wherein a display of the computing device presents the VR/AR controller in the virtual world or augmented-reality viewport.

12. An apparatus comprising:

one or more computer processors;

a radar system comprising;

one or more radar-emitting elements configured to provide a radar field; and

one or more antenna elements configured to receive a radar signal representing a superposition of reflections of two or more points within the radar field provided by the radar system; and

one or more non-transitory computer-readable storage media having instructions stored thereon that, responsive to execution by one or more computer processors, implement a fine-motion tracking module, a VR/AR control module, and a user representation module;

the fine-motion tracking module configured to track the points within the radar field effective to track user actions interacting with a VR/AR controller, the tracked user actions including fine-motions in a millimeter or sub-millimeter range, the tracking comprising;

calculating a radial distance for each of the points using a time delay between a transmittal of the radar field and the receiving of the radar signal;

calculating a radial velocity for each of the points using an observed Doppler frequency for each of the points;

ascertaining a reflected energy for each of the points;

determining a relative displacement, or relative velocity, between the points based on the radial distance, radial velocity, and reflected energy for each of the points, at least one of the points being a portion of a user that performs the user actions;

the VR/AR control module configured to;

present the VR/AR controller within a virtual world; and

alter an orientation of the VR/AR controller in real time and with fine-motion appearance changes, the fine-motion appearance changes in the millimeter or sub-millimeter range and corresponding to the tracked user actions; and

the user representation module configured to present, in the virtual world, the portion of the user performing the user actions interacting with the VR/AR controller.

13. The apparatus of claim 12, wherein the user representation module presents the portion of the user in real time and in the millimeter or sub-millimeter range.

14. The apparatus of claim 13, the user representation module presents the portion of the user showing the portion of the user in contact with the VR/AR controller and in real time.

15. The apparatus of claim 12, wherein the user representation module presents the portion of the user by determining a location in three dimensions for the user actions.

16. The apparatus of claim 12, wherein the fine-motion tracking module is further configured, responsive to the altered orientation of the VR/AR controller corresponding to a control orientation, to pass a control input corresponding to the control orientation effective to control the virtual world.

17. The apparatus of claim 12, wherein the tracking further comprises distinguishing the points by tracking temporal changes in the radar signal.

18. The apparatus of claim 12, wherein the points are on one or more hands of the user.

19. The apparatus of claim 12, wherein the apparatus is proximal a head of the user;

and

wherein the apparatus further comprises a display that presents the VR/AR controller within the virtual world.

20. An apparatus comprising:

one or more computer processors;

a radar system comprising;

one or more radar-emitting elements configured to provide a radar field; and

one or more antenna elements configured to receive a radar signal representing a superposition of reflections of two or more points that are spatially separated within the radar field provided by the radar system; and

one or more non-transitory computer-readable storage media having instructions stored thereon that, responsive to execution by one or more computer processors, implement a fine-motion tracking module and a VR/AR control module;

the fine-motion tracking module configured to track the points within the radar field effective to track user actions interacting with a VR/AR controller, the tracked user actions including fine-motions in a millimeter or sub-millimeter range, the tracking comprising;

distinguishing the spatially separated points by determining respective micro-Doppler centroids for the points; and

determining a relative displacement, or relative velocity, between the points by spatially resolving the points, at least one of the points being a portion of a user that performs the user actions;

the VR/AR control module configured to;

present the VR/AR controller within a viewport for an augmented-reality world; and

alter an orientation of the VR/AR controller in real time and with fine-motion appearance changes, the fine-motion appearance changes in the millimeter or sub-millimeter range and corresponding to the tracked user actions.

21. The apparatus of claim 20, wherein altering the orientation of the VR/AR controller shows a turn or lateral movement of the VR/AR controller, the turn or the lateral movement shown in real time and corresponding to a real motion in a real world of a velocity and displacement of a portion of the user that performs the user actions.

22. The apparatus of claim 20, wherein the VR/AR controller is presented touching the portion of the user in the real world that is performing the user actions.

23. The apparatus of claim 20, responsive to the altered orientation of the VR/AR controller corresponding to a control orientation, passing a control input corresponding to the control orientation effective to control the augmented-reality viewport.

24. The apparatus of claim 20, wherein the spatially resolving is based on:

a radial distance for each of the points using a time delay between transmittal of the radar field and receiving the radar signal,

a radial velocity for each of the points using an observed Doppler frequency, and

a reflected energy for each of the first and second points.

25. The apparatus of claim 20, wherein the points are on one or more hands of the user.

26. The apparatus of claim 20, wherein the apparatus is proximal a head the user;

and

wherein the apparatus further comprises a display that presents the VR/AR controller within the augmented-reality world.