Gesture Component with Gesture Library

• US 20190257939A1
• Filed: 05/02/2019
• Published: 08/22/2019
• Est. Priority Date: 10/06/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Schwesig, Carsten C., Poupyrev, Ivan
• Granted Patent: US 10,768,712 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G01S 13/56 for presence detection presence detection using near field arrangements G01V3/00, e.g. G01V3/08, G01V3/12; burglar, theft or intruder alarms with electrical actuation G08B13/22 - G08B13/26
• CPC Class Codes: G01S 13/66 Radar-tracking systems; Analogous systems
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 13/90 using synthetic aperture techniques , e.g. synthetic aperture radar [SAR] techniques
• CPC Class Codes: G01S 13/904 SAR modes
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4004 of parts of a radar system
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/412 based on a comparison between measured values and known or stored values
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/25 : Fusion techniques
• CPC Class Codes: G06F 18/253 : of extracted features
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 2203/0384 : Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 2221/2105 : Dual mode as a secondary aspect
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/165 : Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/806 : of extracted features
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 17/02 : using a radio link
• CPC Class Codes: G08C 2201/93 : Remote control using other portable devices, e.g. mobile phone, PDA, laptop
• CPC Class Codes: H04Q 2209/883 : where the sensing device enters an active or inactive mode
• CPC Class Codes: H04Q 9/00 : Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
• CPC Class Codes: H04W 16/28 : using beam steering
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

A gesture component with a gesture library is described. The gesture component is configured to expose operations for execution by application of a computing device based on detected gestures. In one example, an input is detected using a three dimensional object detection system of a gesture component of the computing device. A gesture is recognized by the gesture component based on the detected input through comparison with a library of gestures maintained by the gesture component. An operation is then recognized that corresponds to the gesture by the gesture component using the library of gestures. The operation is exposed by the gesture component via an application programming interface to at least one application executed by the computing device to control performance of the operation by the at least one application.

Claims

1. A method of controlling operation of a computing device by radar-based gesture detection, the method comprising:

receiving, by the computing device, raw data representing radar signals that have been reflected off an object;

transforming, by the computing device, the raw data of the radar signals into one or more different representations that represent information about the object;

extracting, by the computing device, three-dimensional feature information about the object from the transformed raw data of the radar signals;

recognizing, by the computing device, a three-dimensional gesture performed by the object, the recognizing based on a comparison of the extracted feature information about the object from the radar signals with a library of gestures maintained by the computing device;

recognizing, by the computing device, an operation that corresponds to the three-dimensional gesture using the library of gestures; and

exposing, by the computing device, an indication of the operation for receipt by at least one application executed by the computing device to cause the at least one application to control performance of the operation.

2. The method of claim 1, wherein the radar signals comprise radio signals.

3. The method of claim 2, wherein the radar signals correspond to a 60 GHz band.

4. The method of claim 1, wherein the at least one application is not made aware as to how the operation is detected by the computing device.

5. The method of claim 1, wherein the library of gestures includes media gestures usable by the at least one application to control rendering of media.

6. The method of claim 1, wherein the library of gestures includes user interface navigation gestures usable by the at least one application to control navigation within a user interface.

7. The method of claim 1, wherein the transformed raw data comprises a complex vector containing phase and amplitude of the radar signals, a spatial representation of the object, a vector containing velocity and direction of the object, or a representation of corresponding frequencies of the radar signals.

8. The method of claim 1, wherein:

the transformed raw data represents a classification of the object such as a hand, a finger, or a head; and

the extracting the three-dimensional feature information about the object comprises looking for three-dimensional feature information specific to the classification of the object.

9. A computing device comprising:

a processing system;

an application maintained in computer-readable storage media and executable by the processing system to perform a plurality of operations; and

a gesture component implemented at least partially in hardware, the gesture component configured to;

receive raw data representing radar signals that have been reflected by an object in three-dimensional space;

transform the raw data of the radar signals into a different representation that represents information about the object;

extract three-dimensional feature information about the object from the transformed raw data of the radar signals;

maintain a library of gestures corresponding to the operations;

compare the three-dimensional feature information about the object from the transformed raw data of the radar signals with the library of gestures;

recognize a gesture performed by the object and a corresponding operation based on the comparison; and

expose an indication of the corresponding operation that corresponds to the recognized gesture for receipt by the application to cause the application to control performance of the corresponding operation.

10. The computing device of claim 9, wherein the radar signals comprise radio waves.

11. The computing device of claim 10, wherein the radio waves correspond to a 60 GHz band.

12. The computing device of claim 9, wherein the application is not made aware as to how the corresponding operation is detected by the gesture component.

13. The computing device of claim 9, wherein:

the operations comprise media control operations; and

the library of gestures includes media gestures corresponding to the media control operations.

14. The computing device of claim 9, wherein:

the operations comprise user interface navigation control operations; and

the library of gestures includes user interface navigation gestures corresponding to the user interface navigation control operations.

15. The computing device of claim 9, wherein the different representation comprises a complex vector containing phase and amplitude of the radar signals, a spatial representation of the object, a vector containing velocity and direction of the object, or a representation of corresponding frequencies of the radar signals.

16. The computing device of claim 9, wherein:

different representation represents a classification of the object such as a hand, a finger, or a head; and

the extracting the three-dimensional feature information about the object comprises looking for three-dimensional feature information specific to the classification of the object.

17. A method performed by a computing device of exposing an operation for an application from radar-based gesture detection, the method comprising:

transmitting, by the computing device, a plurality of outgoing radio frequency radar signals;

capturing, by the computing device, incoming radio frequency radar signals generated by the outgoing radio frequency radar signals reflecting off a user;

exposing, by the computing device, the operation to the application based on the captured incoming radio frequency radar signals, the operation exposed by;

transforming raw data corresponding to the captured incoming radio frequency radar signals into a different representation that represents information about a portion of the user;

extracting three-dimensional feature information about the portion of the user;

comparing the three-dimensional feature information about the portion of the user with a library of gestures maintained by the computing device;

recognizing a gesture performed by the portion of the user and the operation based on the comparison, the operation corresponding to the recognized gesture; and

exposing the operation that corresponds to the recognized gesture to the application such that the application can control performance of the operation.

18. The method of claim 17, wherein the outgoing radio frequency radar signals correspond to a 60 GHz band.

19. The method of claim 17, wherein the operation comprises a media-related operation or a navigation operation.

20. The method of claim 17, wherein:

the portion of the user comprises a body part of the user such as a hand, a finger, or a head; and

the extracting the three-dimensional feature information comprises searching for three-dimensional feature information specific to the body part of the user