Gesture Detection and Interactions

• US 20200150776A1
• Filed: 01/16/2020
• Published: 05/14/2020
• Est. Priority Date: 05/27/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Poupyrev, Ivan, Schwesig, Carsten, Schulze, Jack, Arnall, Timo, Bishop, Durrell Grant Bevington
• Granted Patent: US 10,936,085 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Gesture detection and interaction techniques are described. Object detection used to support the gestures may be accomplished in a variety of ways, such as by using radio waves as part of a radar technique. In a first example, the techniques are implemented such that one hand of a user sets a context for a gesture that is defined by another hand of the user. In another example, a gesture recognition mode is utilized. In yet another example, detection of distance is used such that the same motions may be used to different between operations performed. In a further example, split gestures are supported. In another instance, entry into a gesture recognition mode may be implemented through touch and then recognized through three-dimensional orientation and motion of that hand or another.

Claims

1. A method of controlling operation of a computing device based on three-dimensional gesture detection, the method comprising:

receiving a touch input, by a touchscreen of the computing device, indicating a request from a user of the computing device to enter a gesture recognition mode;

entering the gesture recognition mode based on the touch input; and

responsive to entering the gesture recognition mode;

transmitting, by a three-dimensional object detection system of the computing device, a signal configured to;

pass through an intervening article worn by or associated with the user; and

reflect off human tissue of the user;

receiving, by the three-dimensional object detection system, a reflection of the signal off a portion of the user;

determining, by the computing device, a gesture performed by the portion of the user in three-dimensional space based on the reflection of the signal; and

causing performance of one or more operations, by the computing device, based on the determined gesture.

2. The method of claim 1, wherein one or more of the signal or the reflection did not pass through the intervening article.

3. The method of claim 1, wherein the three-dimensional object detection system uses radar techniques.

4. The method of claim 1, further comprising exiting the gesture recognition mode, by the computing device, responsive to:

causing the performance of the operations;

or

determining that the touch input is no longer being received.

5. The method of claim 4, further comprising, after exiting the gesture recognition mode:

transmitting, by the three-dimensional object detection system, another signal;

receiving, by the three-dimensional object detection system, another reflection of the other signal off the user;

determining, by the computing device, another gesture performed by the user in three-dimensional space based on the other reflection of the other signal; and

ignoring the other gesture based on the computing device not being in the gesture recognition mode.

6. A method of controlling operation of a computing device based on three-dimensional gesture detection, the method comprising:

transmitting, by a three-dimensional object detection system of the computing device, one or more signals configured to;

pass through an intervening article worn by or associated with a user of the computing device; and

reflect off human tissue of the user;

receiving, by the three-dimensional object detection system, a first reflection of a first signal of the signals off a first portion of the user;

determining, by the computing device, a first instance of a first gesture performed by the first portion of the user in three-dimensional space based on the first reflection of the first signal;

receiving, by the three-dimensional object detection system, a second reflection of the first signal or a second signal of the signals off a second portion of the user;

determining, by the computing device, a second gesture performed by the second portion of the user in three-dimensional space based on the second reflection of the first signal or the second signal;

recognizing, by the computing device, the second gesture as an intended gesture based on the determination of the first instance of the first gesture; and

causing performance of one or more operations, by the computing device, based on the second gesture.

7. The method of claim 6, wherein one or more of the first signal, the second signal, the first reflection, or the second reflection did not pass through the intervening article.

8. The method of claim 6, wherein the first gesture is performed by a first hand of the user and the second gesture is performed by a second hand of the user.

9. The method of claim 6:

further comprising storing user-customized reflection data corresponding to the first gesture; and

wherein the first gesture is determined based on a comparison of the first reflection to the user-customized reflection data.

10. The method of claim 6, further comprising:

transmitting, by the three-dimensional object detection system, one or more other signals;

receiving, by the three-dimensional object detection system, a third reflection of a first other signal of the other signals off the user;

determining, by the computing device, a third gesture performed by the user in three-dimensional space based on the third reflection of the first other signal;

failing to determine, by the computing device, a second instance of the first gesture based on reflections of the first other signal or a second other signal of the other signals; and

ignoring the third gesture.

11. A computing device comprising:

a three-dimensional object detection system configured to;

transmit signals configured to;

pass through intervening articles worn by or associated with users of the computing device; and

reflect off human tissue of the users;

receive reflections of the signals reflected off portions of the users; and

determine spatial information about the portions of the users based on the received reflections of the signals; and

a gesture module implemented at least partially in hardware, the gesture module configured to;

receive, from the three-dimensional object detection system, first spatial information about a first portion of the user;

determine a first instance of a first gesture performed by the first portion of the user based on the first spatial information;

receive, from the three-dimensional object detection system, second spatial information about a second portion of the user;

determine a second gesture performed by the second portion of the user based on the second spatial information;

recognize the second gesture as an intended gesture based on the determination of the first instance of the first gesture, and

cause performance of one or more operations of the computing device based on the second gesture.

12. The computing device of claim 11, wherein signals used to determine the first spatial information and the second spatial information did not pass through the intervening article.

13. The computing device of claim 11, wherein the first portion comprises a first hand of the user and the second portion comprises a second hand of the user.

14. The computing device of claim 11, wherein one or more of the first gesture or the second gesture is customized to the user.

15. The computing device of claim 11, wherein the gesture module is further configured to:

receive, from the three-dimensional object detection system, third spatial information about the user;

determine a third gesture performed by the user based on the third spatial information;

fail to determine a second instance of the first gesture from the third spatial information; and

ignore the third gesture.

16. The computing device of claim 11, wherein the gesture module is further configured to:

enter a gesture recognition mode responsive to determining the first gesture; and

exit the gesture recognition mode based on a determination that the first gesture is no longer being performed or that the operations have been completed.

17. The computing device of claim 11, wherein the three-dimensional object detection system uses radar techniques.

18. A method of controlling operation of a computing device based on three-dimensional gesture detection, the method comprising:

displaying, by the computing device, a user interface;

determining, by a three-dimensional object detection system of the computing device, that a portion of a user has entered a predefined threshold distance with the computing device;

determining, by the computing device, a location within a user interface corresponding to a location of the portion of the user relative the user interface when the predefined threshold distance was met;

displaying, by the computing device, an indication of the location on the user interface of the computing device;

receiving, from the three-dimensional object detection system, spatial information about the portion of the user while in the predefined threshold distance; and

changing the indication based on the spatial information;

or

performing an operation on an object of the user interface proximal the indication based on the spatial information.

19. The method of claim 18, further comprising:

determining, by the three-dimensional object detection system, that the portion of the user has exited the predefined threshold distance; and

removing, by the computing device, the indication from the user interface.

20. The method of claim 18, wherein the three-dimensional object detection system uses radar techniques.

21. A method of controlling operation of a computing device based on three-dimensional gesture detection, the method comprising:

transmitting, by a three-dimensional object detection system of the computing device, a first signal configured to reflect off of human tissue;

receiving, by the three-dimensional object detection system, a first reflection of the first signal off of human tissue of a user;

determining, by the computing device, a first gesture performed by the user in three-dimensional space based on the first reflection of the first signal, the first gesture corresponding to a request from the user to enter a gesture recognition mode;

entering the gesture recognition mode based on the determined first gesture; and

responsive to entering the gesture recognition mode;

transmitting, by the three-dimensional object detection system, a second signal;

receiving, by the three-dimensional object detection system, a second reflection of the second signal off of the user;

determining, by the computing device, a second gesture performed by the user in three-dimensional space based on the second reflection of the second signal; and

causing performance of one or more operations, by the computing device, based on the determined second gesture.

22. The method of claim 21, further comprising exiting the gesture recognition mode responsive to causing the performance of the operations.

23. The method of claim 22, further comprising, after exiting the gesture recognition mode:

transmitting, by the three-dimensional object detection system, a third signal;

receiving, by the three-dimensional object detection system, a third reflection of the third signal off of the user;

determining, by the computing device, a third gesture performed by the user in three-dimensional space based on the third reflection of the third signal; and

ignoring the third gesture based on the computing device not being in the gesture recognition mode.