Occluded Gesture Recognition

• US 20190391667A1
• Filed: 09/04/2019
• Published: 12/26/2019
• Est. Priority Date: 08/22/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Poupyrev, Ivan
• Granted Patent: US 10,936,081 B2
• Time in Patent Office: Days
• CPC Class Codes: G01S 13/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S13/46
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G06F 21/316 by observing the pattern of computer usage, e.g. typical user behaviour
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

This document describes techniques and devices for occluded gesture recognition. Through use of the techniques and devices described herein, users may control their devices even when a user'"'"'s gesture is occluded by some material between the user'"'"'s hands and the device itself. Thus, the techniques enable users to control their mobile devices in many situations in which control is desired but conventional techniques do permit effective control, such as when a user'"'"'s mobile computing device is occluded by being in a purse, bag, pocket, or even in another room.

Claims

1. A method performed by a smartphone, the method comprising:

determining that the smartphone is in a location corresponding to a pocket of a user based on sensor data provided by at least one sensor;

providing a radar field emanating through an occlusion from inside the pocket to outside the pocket, the radar field encompassing a likely location where a gesture is to be made with the user'"'"'s hand;

producing an audible sound;

receiving, through the occlusion, a human-tissue reflection of the gesture performed by the user'"'"'s hand;

recognizing, based on the human-tissue reflection, that the gesture is associated with a control input to mute the smartphone; and

muting the audible sound responsive to recognizing the gesture.

2. The method of claim 1, wherein the at least one sensor comprises at least one of the following:

an accelerometer;

an acoustic sensor;

a thermal sensor;

a light sensor;

or

a camera.

3. The method of claim 1, wherein:

the determining that the smartphone is in the pocket comprises determining that the smartphone is in a pocket of the user'"'"'s purse.

4. The method of claim 1, wherein:

the providing of the radar field comprises directing the radar field to encompass the likely location based on the location of the smartphone in the pocket.

5. The method of claim 4, further comprising:

determining an orientation of the smartphone, wherein;

the directing of the radar field is based on both the location of the smartphone and the orientation of the smartphone in the pocket.

6. The method of claim 1, further comprising:

prior to producing the audible sound, tracking the user'"'"'s hand using the radar field.

7. The method of claim 6, further comprising:

adjusting a volume of the radar field based on the tracking of the user'"'"'s hand to conserve power.

8. The method of claim 6, further comprising:

authenticating the user based on authentication credentials;

initiating the tracking of the user'"'"'s hand responsive to authenticating the user; and

identifying the user based on the tracking of the user'"'"'s hand.

9. A smartphone comprising:

a speaker configured to produce an audible sound;

at least one sensor configured to produce sensor data;

an occluded-gesture recognition system configured to;

determine that the smartphone is in a pocket of a user based on the sensor data provided by the at least one sensor;

provide a radar field emanating through an occlusion from inside the pocket to outside the pocket, the radar field encompassing a likely location where a gesture is to be made with the user'"'"'s hand;

receive, through the occlusion, a human-tissue reflection of the gesture performed by the user'"'"'s hand after the speaker produces the audible sound; and

recognize, based on the human-tissue reflection, that the gesture is associated with a control input to mute the smartphone; and

a gesture manager configured to;

mute the audible sound responsive to accepting the control input from the occluded-gesture recognition system.

10. The smartphone of claim 9, wherein the at least one sensor comprises at least one of the following:

an accelerometer;

an acoustic sensor;

a thermal sensor;

a light sensor;

or

a camera.

11. The smartphone of claim 9, wherein:

the occluded-gesture recognition system is further configured to determine that the pocket comprises a pocket on a right side of the user.

12. The smartphone of claim 9, wherein the occluded-gesture recognition system is configured to:

determine an orientation of the smartphone based on the sensor data; and

direct the radar field to encompass the likely location based on both a location of the smartphone corresponding to the pocket and the orientation of the smartphone within the pocket.

13. The smartphone of claim 9, wherein:

the occluded-gesture recognition system is configured to track the user'"'"'s hand using the radar field prior to the speaker producing the audible sound.

14. The smartphone of claim 13, wherein:

the occluded-gesture recognition system is configured to adjust a volume of the radar field based on the tracking of the user'"'"'s hand.

15. The smartphone of claim 13, wherein:

the occluded-gesture recognition system is further configured to initiate tracking of the user'"'"'s hand responsive to the smartphone authenticating the user; and

the gesture manager is configured to identify the user based on the tracking of the user'"'"'s hand.

16. A smartphone comprising:

a speaker configured to produce an audible sound;

at least one sensor configured to produce sensor data;

an occluded-gesture recognition system configured to;

receive, through an occlusion, a human-tissue reflection of a gesture performed by a user'"'"'s hand after the speaker produces the audible sound, the human-tissue reflection occurring at a likely location where a gesture is to be made with the user'"'"'s hand;

one or more computer processors; and

one or more computer-readable storage media having instructions stored thereon that, responsive to execution by the one or more computer processors, perform operations comprising;

determining that the smartphone is in a pocket of the user based on the sensor data;

recognizing, based on the human-tissue reflection, that the gesture is associated with a control input to mute the smartphone; and

muting the audible sound responsive to recognizing the gesture.

17. The smartphone of claim 16, wherein the at least one sensor comprises an accelerometer.

18. The smartphone of claim 16, wherein:

the operation of determining that the smartphone is in the pocket further comprises determining an orientation of the smartphone within the pocket based on the sensor data; and

the occluded-gesture recognition system is configured to determine the likely location where the gesture is to be made based on both a location of the smartphone corresponding to the pocket and the orientation of the smartphone within the pocket.

19. The smartphone of claim 16, wherein:

the operation of determining that the smartphone is in the pocket further comprises determining that the pocket corresponds to a front pocket on the user'"'"'s clothing.

20. The smartphone of claim 16, wherein:

the occluded-gesture recognition system is configured to track the user'"'"'s hand to determine the likely location where the gesture is to be made with the user'"'"'s hand.