Smartphone-based radar system facilitating ease and accuracy of user interactions with displayed objects in an augmented-reality interface

• US 10,698,603 B2
• Filed: 08/24/2018
• Issued: 06/30/2020
• Est. Priority Date: 08/24/2018
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Giusti, Leonardo, Poupyrev, Ivan, Amihood, Patrick M.
• Primary Examiner(s): Pervan, Michael
• Application Number:
  US16/112,130
• Publication Number:
  US 20200064996A1
• Time in Patent Office: 676 Days
• CPC Class Codes: G01S 13/343 using sawtooth modulation
• CPC Class Codes: G01S 13/345 using triangular modulation
• CPC Class Codes: G01S 13/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S13/46
• CPC Class Codes: G01S 13/424 Stacked beam radar
• CPC Class Codes: G01S 13/426 Scanning radar, e.g. 3D radar G01S13/66 takes precedence
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 7/2883 using FFT processing
• CPC Class Codes: G01S 7/356 involving particularities of FFT processing
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10

Abstract

This document describes techniques and systems that enable a smartphone-based radar system facilitating ease and accuracy of user interactions with displayed objects in an augmented-reality interface. The techniques and systems use a radar field to accurately determine three-dimensional (3D) gestures that can be used to interact with augmented-reality (AR) objects that are presented on a display of an electronic device, such as a smartphone. These techniques allow the user to make 3D gestures from a distance—the user does not have to hold the electronic device steady while touching the screen and the gestures do not obstruct the user'"'"'s view of the AR objects presented on the display.

Claims

1. A smartphone, comprising:

a display;

a radar system, implemented at least partially in hardware, configured to;

provide a radar field;

sense reflections from an object in the radar field;

analyze the reflections from the object in the radar field; and

provide, based on the analysis of the reflections, radar data;

one or more computer processors; and

one or more computer-readable media having instructions stored thereon that, responsive to execution by the one or more computer processors, implement a radar-based application configured to;

present an augmented-reality (AR) element through the display of the smartphone, the AR element including a touch input control and related to a real object, an image of which is presented on the display; and

responsive to a determination, based on the radar data, that the object in the radar field is moving toward the display, maintain the touch input control at a fixed location on the display of the smartphone.

2. The smartphone of claim 1, wherein the object in the radar field is a user, and whereby ease and accuracy of the user'"'"'s interaction with the touch input control are facilitated by maintaining the touch input control at the fixed location even as the image of the real object is moving unsteadily on the display.

3. The smartphone of claim 2, wherein the radar-based application is further configured to present the augmented-reality (AR) element, including the touch input control, while the user holds the smartphone with a first hand.

4. The smartphone of claim 3, wherein:

the determination that the object in the radar field is moving toward the display comprises determining that the user is reaching toward the smartphone with a second hand;

the smartphone is positioned such that the image of the real object is no longer being presented on the display; and

the radar-based application is further configured to maintain the AR element and the touch input control at approximately a same location on the display as presented prior to the user moving the smartphone such that the image of the real object is no longer presented on the display.

5. The smartphone of claim 4, wherein:

the touch input control has been activated via a prior touch input by the second hand;

the radar-based application is further configured to present on the display, responsive to the activation of the touch input control, a two-dimensional (2D) interface that includes;

another image of the real object; and

another touch input control that provides the user with information about the real object.

6. The smartphone of claim 1, wherein the radar-based application is further configured to maintain the fixed location of the touch input control when the image of the real object is no longer presented on the display.

7. The smartphone of claim 1, wherein the radar-based application is further configured, responsive to the touch input control at the fixed location not being touched within a threshold time, to stop providing the touch input control at the fixed location.

8. The smartphone of claim 1, wherein the radar-based application is further configured to provide the touch input control at the fixed location if the object in the radar field is moving toward the display at a velocity that exceeds a threshold velocity.

9. The smartphone of claim 1, wherein the smartphone includes an image-capture device and the radar-based application presents, via the image-capture device, the image of the real object in real time or near-real time.

10. The smartphone of claim 1, wherein the radar system is further configured to determine, based on the radar data, that the object in the radar field is moving toward the display.

11. The smartphone of claim 1, wherein the radar-based application is further configured to determine, based on the radar data, that the object in the radar field is moving toward the display.

12. The smartphone of claim 1, wherein the object in the radar field is a body part of a user.

13. A system, comprising:

an electronic device that includes a display;

a radar system, implemented at least partially in hardware, configured to;

provide a radar field;

sense, at a first time, reflections from an object in the radar field;

analyze the reflections from the object in the radar field; and

provide, based on the analysis of the reflections, radar data;

one or more computer processors; and

one or more computer-readable media having instructions stored thereon that, responsive to execution by the one or more computer processors, implement a radar-based application configured to;

present an augmented-reality (AR) element through the display of the electronic device;

receive, at a second time, a touch input made through the display of the electronic device, the touch input effective to select the AR element, the second time being later than the first time;

determine, based on the radar data, a gesture by the object in the radar field; and

perform an action related to the selected AR element, the action corresponding to the determined gesture.

14. The system of claim 13, wherein the determined gesture is a three-dimensional (3D) gesture.

15. The system of claim 13, wherein the object is a body part of the user.

16. The system of claim 13, wherein the determined gesture comprises changing a distance between the object and the radar system.

17. The system of claim 16, wherein:

the action corresponding to the changed distance being an increased distance between the object and the radar system is a movement of the selected AR element closer to the user; and

the action corresponding to the changed distance being a decreased distance between the object and the radar system is a movement of the selected AR element away from the user.

18. The system of claim 17, wherein:

the movement of the selected AR element closer to the user is proportional to the increased distance; and

the movement of the selected AR element away from the user is proportional to the decreased distance.

19. The system of claim 13, wherein the determined gesture comprises changing a position of the object, relative to the radar system, while maintaining a substantially similar distance between the body part and a plane of the display of the electronic device.

20. The system of claim 19, wherein:

the action corresponding to changing the position in a first direction is a rotation of the selected AR element in a first rotational direction about an axis of the selected AR element; and

the action corresponding to changing the position in a second direction is a rotation of the selected AR element in a second rotational direction about the axis.

21. The system of claim 13, wherein the radar system further comprises a digital beamformer and an angle estimator, and the radar system is configured to monitor angles in a field of view between approximately −

90 degrees and approximately 90 degrees.

22. A method implemented in an electronic device that includes a display, a radar system, and a radar-based application, the method comprising:

providing, by the radar system, a radar field;

sensing, by the radar system, reflections from an object in the radar field;

analyzing the reflections from the object in the radar field;

providing, based on the analysis of the reflections, radar data;

presenting, by the radar-based application, an augmented-reality (AR) element through the display of the electronic device, the AR element including a touch input control and related to a real object, an image of which is presented on the display; and

responsive to a determination, based on the radar data, that the object in the radar field is moving toward the display, maintaining the touch input control at a fixed location on the display.

23. The method of claim 22, wherein the determination, based on the radar data, that the object in the radar field is moving toward the display is made by the radar system or by the radar-based application.

24. The method of claim 22, wherein the electronic device is a handheld device, the object in the radar field is a user, and the method further comprises:

maintaining the touch input control at the fixed location even as the image of the real object is moving unsteadily on the display, whereby ease and accuracy of the user'"'"'s interaction with the touch input control are facilitated.

25. The method of claim 24, further comprising:

presenting, by the radar-based application, the augmented-reality (AR) element, including the touch input control, while the user holds the electronic device with a first hand.

26. The method of claim 25, wherein:

the determination that the object in the radar field is moving toward the display comprises determining that the user is reaching toward the electronic device with a second hand;

the electronic device is positioned such that the image of the real object is no longer being presented on the display; and

the method further comprises;

maintaining, by the radar-based application, the AR element and the touch input control at approximately a same location on the display as presented prior to the user moving the electronic device such that the image of the real object is no longer presented on the display.

27. The method of claim 26, wherein the touch input control has been activated via a prior touch input by the second hand, and the method further comprises:

responsive to the activation of the touch input control, presenting on the display, by the radar-based application, a two-dimensional (2D) user interface, the 2D user interface including;

another image of the real object; and

another touch input control that provides the user with information about the real object.