Gesture Component with Gesture Library

• US 20190011989A1
• Filed: 10/05/2016
• Published: 01/10/2019
• Est. Priority Date: 10/06/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Schwesig, Carsten C., Poupyrev, Ivan
• Granted Patent: US 10,379,621 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G01S 13/56 for presence detection presence detection using near field arrangements G01V3/00, e.g. G01V3/08, G01V3/12; burglar, theft or intruder alarms with electrical actuation G08B13/22 - G08B13/26
• CPC Class Codes: G01S 13/66 Radar-tracking systems; Analogous systems
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 13/90 using synthetic aperture techniques , e.g. synthetic aperture radar [SAR] techniques
• CPC Class Codes: G01S 13/904 SAR modes
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4004 of parts of a radar system
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/412 based on a comparison between measured values and known or stored values
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/25 : Fusion techniques
• CPC Class Codes: G06F 18/253 : of extracted features
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 2203/0384 : Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 2221/2105 : Dual mode as a secondary aspect
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/165 : Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/806 : of extracted features
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 17/02 : using a radio link
• CPC Class Codes: G08C 2201/93 : Remote control using other portable devices, e.g. mobile phone, PDA, laptop
• CPC Class Codes: H04Q 2209/883 : where the sensing device enters an active or inactive mode
• CPC Class Codes: H04Q 9/00 : Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
• CPC Class Codes: H04W 16/28 : using beam steering
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

A gesture component with a gesture library is described. The gesture component is configured to expose operations for execution by application of a computing device based on detected gestures. In one example, an input is detected using a three dimensional object detection system of a gesture component of the computing device. A gesture is recognized by the gesture component based on the detected input through comparison with a library of gestures maintained by the gesture component. An operation is then recognized that corresponds to the gesture by the gesture component using the library of gestures. The operation is exposed by the gesture component via an application programming interface to at least one application executed by the computing device to control performance of the operation by the at least one application.

Claims

1. A method of controlling operation of a computing device based on gesture detection, the method comprising:

detecting an input using a three dimensional object detection module of a gesture component of the computing device;

transforming raw data of the detected input into one or more different representations that represent information about an object used to provide the input;

extracting three dimensional feature information about the object from the transformed raw data;

recognizing a gesture performed by the object by the gesture component based on the detected input through comparison of the extracted feature information about the object with a library of gestures maintained by the gesture component;

recognizing an operation that corresponds to the gesture by the gesture component using the library of gestures; and

exposing an indication of the operation by the gesture component via an application programming interface for receipt by at least one application executed by the computing device to cause the at least one application to control performance of the operation.

2. The method of claim 1, wherein the three dimensional object detection system performs the detecting using radar techniques involving radio signals.

3. The method of claim 2, wherein the radio signals correspond to a frequency band included as part of a Wi-Fi radio spectrum.

4. The method of claim 1, wherein the at least one application is not made aware through the application program interface as to how the operation is detected by the three dimensional object detection module.

5. The method of claim 1, wherein the library of gestures includes media gestures usable by the at least one application to control rendering of media.

6. The method of claim 1, wherein the library of gestures includes user interface navigation gestures usable by the at least one application to control navigation within a user interface.

8. The method as described in claim 1, wherein the three dimensional object detection module and the gesture library of the gesture component are implemented via a single integrated circuit.

21. The method of claim 1, wherein the information about the object comprises a classification of the object such as a hand, a finger, or a head;

and

wherein the extracting the three dimensional feature information about the object comprises looking for three dimensional feature information specific to the classification of the object.

7. (canceled)

9. A computing device comprising:

a processing system;

an application maintained in computer-readable storage media and executable by the processing system to perform a plurality of operations; and

a gesture component implemented at least partially in hardware, the gesture component including;

a three dimensional object detection module configured to detect an input involving orientation or movement of an object in three dimensional space; and

a gesture module configured to;

maintain a library of gestures;

transform raw data corresponding to the input into one or more different representations that represent information about the object;

extract three dimensional feature information about the object from the transformed raw data;

recognize a gesture performed by the object, and from the gesture, at least one of the plurality of operations that correspond to the gesture based on the detected input through comparison of the extracted feature information about the object with the library of gestures; and

expose an indication of at least one said operation via an application programming interface for receipt by at least one application to cause the at least one application to control performance of the at least one said operation.

10. The computing device of claim 9, wherein the three dimensional object detection system detects the input using radar techniques involving radio signals.

11. The computing device of claim 10, wherein the radio signals correspond to a frequency band included as part of a Wi-Fi radio spectrum.

12. The computing device of claim 9, wherein the at least one application is not made aware through the application program interface as to how the at least one said operation is detected by the three dimensional object detection module.

13. The computing device of claim 9, wherein the library of gestures includes media gestures usable by the at least one application to control output of media.

14. The computing device of claim 9, wherein the library of gestures includes user interface navigation gestures usable by the at least one application to control navigation within a user interface.

16. The computing device of claim 9, wherein the three dimensional object detection module and the gesture library of the gesture component are implemented via a single integrated circuit.

22. The computing device of claim 9, wherein the information about the object comprises a classification of the object such as a hand, a finger, or a head;

and

wherein the extracting the three dimensional feature information about the object comprises looking for three dimensional feature information specific to the classification of the object.

15. (canceled)

17. A method of controlling operation of a computing device based on gesture detection, the method comprising:

transmitting a plurality of outgoing radio frequency signals by a gesture component;

capturing incoming radio frequency signals generated by the plurality of outgoing radio frequency signals as reflecting off a user;

generating an input by the gesture component from the captured incoming radio frequency signals, the input generated by;

transforming raw data corresponding to the incoming radio frequency signals into one or more different representations that represent information about one or more portions of the user;

extracting three dimensional feature information about the one or more portions of the user;

recognizing a gesture performed by the one or more portions of the user and at least one operation of a plurality of operations that correspond to the gesture by the gesture component based on the input through comparison of the extracted feature information about the one or more portions of the user with a library of gestures maintained by the gesture component; and

exposing an indication of the at least one operation by the gesture component via an application programming interface for receipt by at least one application executed by the computing device to cause the at least one application to control performance of the at least one operation.

18. The method of claim 17, wherein the radio signals correspond to a frequency band included as part of a Wi-Fi radio spectrum.

19. The method of claim 17, wherein the library of gestures includes media gestures usable by the at least one application to control output of media or user interface navigation gestures usable by the at least one application to control navigation within a user interface.

23. The method of claim 17, wherein the information about the one or more portions of the user comprises one or more body parts of the user such as a hand, a finger, or a head;

and

wherein the extracting the three dimensional feature information about the one or more portions of the user comprises looking for three dimensional feature information specific to the one or more body parts of the user.

20. (canceled)