Devices, systems, and methods for detecting gestures using wireless communication signals

• US 9,971,414 B2
• Filed: 04/01/2014
• Issued: 05/15/2018
• Est. Priority Date: 04/01/2013
• Status: Active Grant

• Current Assignee: University of Washington
• Original Assignee: University of Washington
• Inventors: Gollakota, Shyamnath, Patel, Shwetak N., Pu, Qifan, Gupta, Sidhant
• Primary Examiner(s): Merkoulova, Olga
• Assistant Examiner(s): Suteerawongsa, Jarurat
• Application Number:
  US14/779,946
• Publication Number:
  US 20160054804A1
• Time in Patent Office: 1,505 Days
• CPC Class Codes: G06F 13/28 using burst mode transfer, e.g. direct memory access DMA, cycle steal G06F13/32 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Examples of systems, devices, and methods are described herein that can provide for gesture recognition. Wireless communication signals are received from sources in an environment (e.g. cellular telephones, computers, etc.). Features of the wireless communication signals (e.g. Doppler shifts) are extracted and utilized to identify gestures. The use of wireless communication signals accordingly make possible gesture recognition in a whole-home environment that identifies gestures performed through walls or other obstacles.

Claims

1. A device comprising:

a receiver configured to receive wireless communication signals, each wireless communication signal received from a plurality of sub-channels;

at least one processing unit configured to identify features in the wireless communication signals, wherein identifying the features in the wireless communication signals comprises;

equalizing received symbols in each sub-channel of the plurality of sub-channels, wherein equalizing the received symbols comprises multiplying each received symbol in a respective sub-channel with a ratio including a first received symbol of the respective sub-channel to provide equalized symbols;

transforming the equalized symbols with a Fast Fourier Transform (FFT) to provide a narrowband signal; and

analyzing a Doppler profile of the narrowband signal based on the equalized symbols to identify the features in the wireless communication signals;

a memory accessible to the at least one processing unit and storing information regarding features in wireless communication signals caused by gestures; and

wherein the at least one processing unit is further configured to identify a gesture based on the identified features in the wireless communication signals.

2. The device of claim 1, wherein the gestures include two-dimensional and three-dimensional gestures.

3. The device of claim 1, wherein the wireless communication signals include Orthogonal Frequency-Division Multiplexing (OFDM) signals, wherein the received symbols correspond to OFDM symbols.

4. The device of claim 1, wherein the wireless communication signals includes received signal strength indicators (RSSI), phase information, or combinations thereof.

5. The device of claim 1, wherein the wireless communication signals are transmitted from a cellular telephone, computer, or wireless router.

6. The device of claim 1, wherein identifying features in the wireless communication signals further comprises extracting a plurality of Doppler shifts in the Doppler profile.

7. The device of claim 6, wherein identifying features in the wireless communication signals further comprises:

identifying a set of segments in the plurality of Doppler shifts; and

classifying the set of segments as either positive Doppler shifts, negative Doppler shifts, or positive/negative Doppler shifts.

8. The device of claim 7, wherein classifying the set of segments as either positive Doppler shifts, negative Doppler shifts, or positive/negative Doppler shifts comprises:

correlating the set of segments with a gesture signature, the gesture signature indicative of a known set of segments.

9. The device of claim 1 wherein the wireless communication signals comprise beacon signals.

10. The device of claim 1 wherein the at least one processing unit is configured to identify a preamble gesture from the features in the communication signals, and wherein the at least one processing unit is further configured to identify when a subsequent gesture is made by a same entity performing the preamble gesture.

11. The device of claim 1, further comprising a transmitter configured to transmit a control signal responsive to identification of the gesture.

12. The device of claim 1, wherein equalizing the received symbols in each sub-channel of the plurality of sub-channels according to the first received symbol of each respective sub-channel comprises modifying each received symbol using the respective first received symbol of each respective sub-channel.

13. A method comprising:

receiving wireless communication signals from devices in an environment, each wireless communication signal received from a plurality of sub-channels;

detecting features in the wireless communication signals, wherein detecting the features in the wireless communication signals comprises;

equalizing received symbols in each sub-channel of the plurality of sub-channels, wherein equalizing the received symbols comprises multiplying each received symbol in a respective sub-channel with a ratio including a first received symbol of the respective sub-channel to provide equalized symbols;

transforming the equalized symbols with a Fast Fourier Transform (FFT) to provide a narrowband signal; and

analyzing a Doppler profile of the narrowband signal based on the equalized symbols to identify the features in the wireless communication signals;

identifying a gesture based, at least in part, on the features in the wireless communication signals; and

providing a control signal responsive to identification of the gesture.

14. The method of claim 13, wherein the wireless communication signals are Orthogonal Frequency Division Multiplexing (OFDM) signals.

15. The method of claim 13, wherein said detecting features in the wireless communication signals further comprises identifying at least one Doppler shift in the Doppler profile of the narrowband signal.

16. The method of claim 15, wherein identifying a gesture based, at least in part, on the features in the wireless communication signals comprises comparing the features to a plurality of stored gesture signatures.

17. The method of claim 13, wherein said identifying a gesture comprises correlating the features with a database of stored features related to gestures.

18. The method of claim 17, wherein said correlating comprises using machine learning, pattern matching, or combinations thereof.

19. The method of claim 13, wherein the wireless communication signals are provided by a cellular telephone, computer, router, or combinations thereof.

20. A method, comprising:

receiving wireless communication signals from devices in an environment;

detecting features in the wireless communication signals;

identifying a gesture based, at least in part, on the features in the wireless communication signals, wherein the gesture comprises performance of a preamble gesture, wherein identifying the gesture based, at least in part, on the features in the wireless communication signals comprises;

equalizing received symbols in each sub-channel of the plurality of sub-channels to provide equalized symbols, wherein equalizing the received symbols comprises first received symbol of the respective sub-channel to provide equalized symbols;

transforming the equalized symbols with a Fast Fourier Transform (FFT) to provide a narrowband signal; and

comparing a Doppler profile of the narrowband signal with the plurality of gesture signatures to identify the gesture;

determining that a Multiple-Input Multiple-Output (MIMO) channel of a plurality of MIMO channels maximizes Doppler energy from an entity associated with the preamble gesture;

tracking the entity associated with the preamble gesture with the MIMO channel;

identifying a subsequent gesture based, at least in part, on the features in additional wireless communication signals from the tracked entity associated with the preamble gesture; and

providing a control signal responsive to identification of the subsequent gesture.

21. The method of claim 20, further comprising detecting further features in the wireless communication signals and determining whether the further features occurred due to an activity of the entity.

22. A system comprising:

a wireless communications device configured to provide periodic beacon signals using an Orthogonal Frequency Division Multiplexing (OFDM) protocol, each periodic beacon signal received from a plurality of sub-channels;

a router configured to receive the periodic beacon signals from the wireless communications device, wherein the router is in communication with a memory storing a plurality of gesture signatures, and wherein the router is configured to detect a gesture based on a comparison of features in the periodic beacon signals from the wireless communications device and the gesture signatures, wherein detecting the gesture comprises;

equalizing received symbols in each sub-channel of the plurality of sub-channels to provide equalized symbols, wherein equalizing the received symbols comprises multiplying each received symbol in a respective sub-channel with a ratio including a first received symbol of the respective sub-channel to provide equalized symbols;

transforming the equalized symbols with a Fast Fourier Transform (FFT) to provide a narrowband signal; and

comparing a Doppler profile of the narrowband signal with the plurality of gesture signatures to identify the gesture,

wherein the router further includes a transmitter configured to transmit a control signal responsive to the gesture; and

a controller coupled to a power outlet and including a receiver configured to receive the control signal, wherein the controller is configured to turn power on, off, or combinations thereof, responsive to the control signal.

23. The system of claim 22, further including a plurality of wireless communications devices, and wherein the router is further configured to receive the periodic beacon signals from the plurality of wireless communications devices.

24. The system of claim 22, wherein the router is further configured to identify a preamble gesture and track a user having made the preamble gesture.

25. The system of claim 24, wherein the router is further configured to differentiate between gestures made by the user and other entities.