Gesture recognition using multiple antenna

• US 10,817,065 B1
• Filed: 04/07/2016
• Issued: 10/27/2020
• Est. Priority Date: 10/06/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Lien, Jaime, Gillian, Nicholas Edward, Poupyrev, Ivan
• Primary Examiner(s): Sadio, Insa
• Application Number:
  US15/093,533
• Time in Patent Office: 1,664 Days
• Field of Search: 345158, 715863, 715806
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G01S 13/56 for presence detection presence detection using near field arrangements G01V3/00, e.g. G01V3/08, G01V3/12; burglar, theft or intruder alarms with electrical actuation G08B13/22 - G08B13/26
• CPC Class Codes: G01S 13/66 Radar-tracking systems; Analogous systems
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 13/90 using synthetic aperture techniques , e.g. synthetic aperture radar [SAR] techniques
• CPC Class Codes: G01S 13/904 SAR modes
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4004 of parts of a radar system
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/412 based on a comparison between measured values and known or stored values
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/25 : Fusion techniques
• CPC Class Codes: G06F 18/253 : of extracted features
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 2203/0384 : Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 2221/2105 : Dual mode as a secondary aspect
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/165 : Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/806 : of extracted features
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 17/02 : using a radio link
• CPC Class Codes: G08C 2201/93 : Remote control using other portable devices, e.g. mobile phone, PDA, laptop
• CPC Class Codes: H04Q 2209/883 : where the sensing device enters an active or inactive mode
• CPC Class Codes: H04Q 9/00 : Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
• CPC Class Codes: H04W 16/28 : using beam steering
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Various embodiments wirelessly detect micro gestures using multiple antenna of a gesture sensor device. At times, the gesture sensor device transmits multiple outgoing radio frequency (RF) signals, each outgoing RF signal transmitted via a respective antenna of the gesture sensor device. The outgoing RF signals are configured to help capture information that can be used to identify micro-gestures performed by a hand. The gesture sensor device captures incoming RF signals generated by the outgoing RF signals reflecting off of the hand, and then analyzes the incoming RF signals to identify the micro-gesture.

Claims

1. A device configured to identify a micro-gesture associated with a target object, the device comprising:

at least two antennas to transmit a plurality of outgoing radio frequency (RF) signals, each antenna configured to;

transmit a respective outgoing radio frequency (RF) signal of the plurality of outgoing RF signals; and

receive an incoming RF signal generated by at least one transmitted outgoing RF signal of the plurality of outgoing RF signals reflecting off the target object;

a digital signal processing component configured to;

process a first set of data originating from incoming RF signals to extract information about the target object, the information comprising a shape of the object corresponding to at least two points in time; and

a machine-learning component configured to;

receive the information extracted by the digital signal processing component; and

process the information to identify the micro-gesture.

2. The device as recited in claim 1, wherein the digital signal processing component is further configured to transform digital samples of incoming RF signals into a 3-dimensional (3D) spatial profile of the target object.

3. The device as recited in claim 2, wherein the machine-learning component is further configured to process the information using a Random Forrest algorithm.

4. The device as recited in claim 1, wherein the digital signal processing component is further configured to determine transmission parameters for each respective outgoing RF signal effective to transmit a specific diversity scheme.

5. The device as recited in claim 4, wherein the specific diversity scheme comprises a beamforming scheme.

6. The device as recited in claim 1, wherein the target object comprises a hand.

7. The device as recited in claim 1 further comprising an output logic component configured to:

receive multiple input notifications, each input notification indicating a respective instance of the micro-gesture that has been identified; and

filter the multiple input notifications into one output notification.

8. The device as recited in claim 1 further configured to:

receive at least one input configuration parameter; and

modify, based upon the at least one input configuration parameter, a transmission parameter of at least one respective outgoing radio frequency (RF) signal.

9. The device as recited m claim 1, wherein the device comprises a System-on-Chip (SoC).

10. A method for identifying a micro-gesture performed by a hand, the method comprising:

transmitting a plurality of outgoing RF signals, each outgoing RF signal transmitted on a respective antenna of a plurality of antennas;

capturing, at least two incoming RF signals generated by at least two outgoing RF signals of the plurality of outgoing RF signals reflecting off the hand; and

processing the at least two captured incoming RF signals to identify the micro-gesture performed by the hand, the micro-gesture comprising a movement on a millimeter scale.

11. The method as recited in claim 10, wherein the processing comprises:

extracting information from each respective RF signal of the at least two captured incoming RF signals; and

processing the information extracted from each respective RF signal to generate a 3-dimensional (3D) spatial profile of the hand.

12. The method as recited in claim 10 further comprising:

configuring the plurality of outgoing radio frequency (RF) signals based, at least in part, on an expected receive-side algorithm used to process the at least two incoming RF signals.

13. The method as recited in claim 10, wherein the processing comprises:

extracting information from each respective RF signal of the at least two captured incoming RF signals, the information comprising at least one of;

velocity information associated with the target object;

orientation information associated with the target object;

directional movement information associated with the target object;

shape information associated with the target object;

or

size information associated with the target object.

14. The method as recited in claim 10, the processing the at least two captured incoming RF signals to identify the micro-gesture further comprising:

identifying a portion of the hand as being stationary; and

identifying a portion of the hand as having movement.

15. The method as recited in claim 10, the processing the at least two captured incoming RF signals further comprising:

extracting data from the at least two captured incoming RF signals to generate a first level of information that classifies a target object as the hand;

extracting, from the first level of information, a second level of information that identifies the micro-gesture from a plurality of hand-related micro-gestures.

16. A device for detecting a micro-gesture performed by a hand, the device comprising:

a gesture sensor component comprising;

at least two antenna, each respective antenna associated with a respective transceiver path;

at least one processor;

one or more computer-readable storage devices; and

one or more Application Programming Interfaces (APis) stored on the one or more computer-readable storage devices which, responsive to execution by the at least one processor, configure the gesture sensor component to detect the micro-gesture by causing the gesture sensor component to perform operations comprising;

transmitting a plurality of outgoing radio frequency (RF) signals, each outgoing radio frequency (RF) signal being transmitted on a respective antenna of the at least two antenna;

receiving at least two incoming RF signals originating from at least two outgoing RF signals of the plurality of outgoing RF signals reflecting off the hand; and

processing the at least two RF signals to detect the micro-gesture by processing data from at least two respective transceiver paths, the micro-gesture comprising a relative movement between a portion of the hand and another portion of the hand.

17. The device as recited in claim 16, wherein at least one API of the one or more APIs comprises one of:

an API to configure transmission parameters associated with the plurality of outgoing RF signals;

an API to configure a digital signal processing component that performs, at least in part, the processing the at least two RF signals;

or

an API to configure a machine-learning component that performs, at least in part, the processing the at least two RF signals.

18. The device as recited in claim 16, wherein at least one API of the one or more APIs is configured to enable registration for an event notification when the micro-gesture is detected.

19. The device as recited in claim 16, wherein processing the at least two RF signals to detect the micro-gesture further comprises utilizing a pipeline to extract information from the at least two RF signals at different stages of the pipeline, and detect the micro-gesture using the information extracted from the at least two RF signals.

20. The device as recited in claim 16, wherein the processing the at least two RF signals further comprises processing the at least two RF signals using a short-range object detection mode.