Fine-Motion Virtual-Reality or Augmented-Reality Control Using Radar

• US 20200089314A1
• Filed: 11/20/2019
• Published: 03/19/2020
• Est. Priority Date: 10/06/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Poupyrev, Ivan, Amihood, Patrick M.
• Granted Patent: US 11,256,335 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G01S 13/56 for presence detection presence detection using near field arrangements G01V3/00, e.g. G01V3/08, G01V3/12; burglar, theft or intruder alarms with electrical actuation G08B13/22 - G08B13/26
• CPC Class Codes: G01S 13/66 Radar-tracking systems; Analogous systems
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 13/90 using synthetic aperture techniques , e.g. synthetic aperture radar [SAR] techniques
• CPC Class Codes: G01S 13/904 SAR modes
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4004 of parts of a radar system
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/412 based on a comparison between measured values and known or stored values
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/25 : Fusion techniques
• CPC Class Codes: G06F 18/253 : of extracted features
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 2203/0384 : Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 2221/2105 : Dual mode as a secondary aspect
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/165 : Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/806 : of extracted features
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 17/02 : using a radio link
• CPC Class Codes: G08C 2201/93 : Remote control using other portable devices, e.g. mobile phone, PDA, laptop
• CPC Class Codes: H04Q 2209/883 : where the sensing device enters an active or inactive mode
• CPC Class Codes: H04Q 9/00 : Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
• CPC Class Codes: H04W 16/28 : using beam steering
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

This document describes techniques for fine-motion virtual-reality or augmented-reality control using radar. These techniques enable small motions and displacements to be tracked, even in the millimeter or sub-millimeter scale, for user control actions even when those actions are small, fast, or obscured due to darkness or varying light. Further, these techniques enable fine resolution and real-time control, unlike conventional RF-tracking or optical-tracking techniques.

Claims

1. A computer-implemented method for fine-motion virtual-reality (VR) or augmented-reality (AR) control, the computer-implemented method comprising:

presenting a VR/AR object within a VR world or an AR viewport;

tracking a user interaction with the VR/AR object, the user interaction including a fine-motion in a millimeter or sub-millimeter range, the tracking comprising;

receiving a radar signal representing a superposition of reflections of two or more spatially separated points over time within a radar field provided by a radar system;

calculating a first radial distance for a first point of the points at a first time based on the received radar signal;

calculating a first radial velocity for the first point at the first time using an observed Doppler frequency for the first point at the first time;

calculating a second radial distance for the first point at a second time based on the received radar signal;

calculating a second radial velocity for the first point at the second time using an observed Doppler frequency for the first point at the second time;

determining a movement of the first point between the first and second times based on the first and second radial distances and first and second radial velocities; and

establishing the user interaction based on the determined movement of the first point;

altering the VR/AR object in real time corresponding to the user interaction; and

passing a control input corresponding to the user interaction effective to control the VR world, the AR viewport, or an application or device external to the VR world or the AR viewport.

2. The computer-implemented method of claim 1, wherein:

the VR/AR object comprises a number pad or virtual keyboard; and

the user interaction is a button press on the number pad or the virtual keyboard.

3. The computer-implemented method of claim 1, wherein the method is performed by smart glasses or VR goggles.

4. The computer-implemented method of claim 1, wherein:

the VR/AR object is a button, knob, or slider; and

the user interaction is a press of the button, a turn of the knob, or a movement of the slider.

5. The computer-implemented method of claim 1, wherein:

the VR/AR object is a steering wheel; and

the user interaction is a turn of the steering wheel.

6. The computer-implemented method of claim 1, wherein the user interaction is a selection corresponding to the VR/AR object.

7. The computer-implemented method of claim 1, wherein the VR/AR object is superimposed on a real world object such that the user interaction corresponds to the real world object.

8. The computer-implemented method of claim 1, wherein the radar signal is received through fabric.

9. The computer-implemented method of claim 8, wherein the method is performed by a computing device that is disposed in a pocket or bag of a user performing the user interaction.

10. The computer-implemented method of claim 1, wherein a portion of a user that performs the user interaction is viewable by the user through the AR viewport.

11. The computer-implemented method of claim 10, wherein the portion of the user is one or more hands of the user.

12. The computer-implemented method of claim 1, further comprising displaying a visual indication of the user interaction with the VR/AR object in real time corresponding to the user interaction.

13. The computer-implemented method of claim 12, wherein the displaying the visual indication comprises altering an appearance of the VR/AR object.

14. The computer-implemented method of claim 13, wherein the altering the appearance of the VR/AR object comprises changing a color of a portion of the VR/AR object.

15. The computer-implemented method of claim 1, wherein:

the tracking further comprises;

calculating a first radial distance for a second point of the points at the first time based on the received radar signal;

calculating a first radial velocity for the second point at the first time using an observed Doppler frequency for the second point at the first time;

calculating a second radial distance for the second point at the second time based on the received radar signal;

calculating a second radial velocity for the second point at the second time using an observed Doppler frequency for the second point at the second time; and

determining a movement of the second point between the first and second times based on the first and second radial distances and first and second radial velocities for the second point; and

the establishing the user interaction is further based on the determined movement of the second point.

16. The computer-implemented method of claim 15, wherein the first point is on a hand of a user that performs the user interaction and the second point is not on a portion of the user.

17. The computer-implemented method of claim 15, wherein the first and second points are on a hand of a user that performs the user interaction.

18. The computer-implemented method of claim 17, wherein the first point is on a first finger of the hand of the user and the second point is on a second finger of the hand of the user.

19. The computer-implemented method of claim 18, wherein the user interaction comprises the first finger crossing the second finger.

20. The computer-implemented method of claim 15, wherein the first point is visually obscured from a computing device that performs the method.

21. The computer-implemented method of claim 19, wherein the first point is visually obscured by the second point.

22. A computer-implemented method for fine-motion virtual-reality (VR) or augmented-reality (AR) control, the computer-implemented method comprising:

presenting a VR/AR object within a VR world or an AR viewport;

tracking a user interaction with the VR/AR object, the user interaction including a fine-motion in a millimeter or sub-millimeter range, the tracking comprising;

receiving a radar signal representing a superposition of reflections of two or more spatially separated points over time within a radar field provided by a radar system;

distinguishing the spatially separated points by determining respective Doppler centroids for the points;

determining a movement of a first point of the points between first and second times based on spatially resolving the first point at the first and second times; and

establishing the user interaction based on the determined movement of the first point;

altering the VR/AR object in real time corresponding to the user interaction; and

passing a control input corresponding to the user interaction effective to control the VR world, the AR viewport, or an application or device external to the VR world or the AR viewport.

23. The computer-implemented method of claim 22, wherein:

the VR/AR object comprises a number pad or virtual keyboard; and

the user interaction is a button press on the number pad or the virtual keyboard.

24. The computer-implemented method of claim 22, wherein the method is performed by smart glasses or VR goggles.

25. The computer-implemented method of claim 22, wherein:

the VR/AR object is a button, knob, or slider; and

the user interaction is a press of the button, a turn of the knob, or a movement of the slider.

26. The computer-implemented method of claim 22, wherein:

the VR/AR object is a steering wheel; and

the user interaction is a turn of the steering wheel.

27. The computer-implemented method of claim 22, wherein the user interaction is a selection corresponding to the VR/AR object.

28. The computer-implemented method of claim 22, wherein the VR/AR object is superimposed on a real world object such that the user interaction corresponds to the real world object.

29. The computer-implemented method of claim 22, wherein the radar signal is received through fabric.

30. The computer-implemented method of claim 29, wherein the method is performed by a computing device that is disposed in a pocket or bag of a user performing the user interaction.

31. The computer-implemented method of claim 22, wherein a portion of a user that performs the user interaction is viewable by the user through the AR viewport.

32. The computer-implemented method of claim 31, wherein the portion of the user is one or more hands of the user.

33. The computer-implemented method of claim 22, further comprising displaying a visual indication of the user interaction with the VR/AR object in real time corresponding to the user interaction.

34. The computer-implemented method of claim 33, wherein the displaying the visual indication comprises altering an appearance of the VR/AR object.

35. The computer-implemented method of claim 34, wherein the altering the appearance of the VR/AR object comprises changing a color of a portion of the VR/AR object.

36. The computer-implemented method of claim 22, wherein:

the tracking further comprises determining a movement of a second point of the points between the first and second times based on spatially resolving the second point at the first and second times; and

the establishing the user interaction is further based on the determined movement of the second point.

37. The computer-implemented method of claim 36, wherein the first point is on a hand of a user that performs the user interaction and the second point is not on a portion of the user.

38. The computer-implemented method of claim 36, wherein the first and second points are on a hand of a user that performs the user interaction.

39. The computer-implemented method of claim 38, wherein the first point is on a first finger of the hand of the user and the second point is on a second finger of the hand of the user.

40. The computer-implemented method of claim 39, wherein the user interaction comprises the first finger crossing the second finger.

41. The computer-implemented method of claim 36, wherein the first point is visually obscured from a computing device that performs the method.

42. The computer-implemented method of claim 41, wherein the first point is visually obscured by the second point.