Device-free tracking system that accurately tracks hand movement

• US 10,386,482 B2
• Filed: 01/24/2017
• Issued: 08/20/2019
• Est. Priority Date: 01/25/2016
• Status: Active Grant

• Current Assignee: Board of Regents of the University of Texas System (University of Texas System)
• Original Assignee: Board of Regents of the University of Texas System (University of Texas System)
• Inventors: Qiu, Lili, Yun, Sangki
• Primary Examiner(s): Pihulic, Daniel
• Application Number:
  US15/414,084
• Publication Number:
  US 20170212235A1
• Time in Patent Office: 938 Days
• CPC Class Codes: G01S 15/34 using transmission of continuous, frequency-modulated waves while heterodyning the received signal, or a signal derived therefrom, with a locally-generated signal related to the contemporaneously transmitted signal
• CPC Class Codes: G01S 15/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S15/46
• CPC Class Codes: G01S 15/46 Indirect determination of position data
• CPC Class Codes: G01S 15/582 using transmission of interrupted pulse-modulated waves and based upon the Doppler effect resulting from movement of targets
• CPC Class Codes: G01S 15/586 using transmission of continuous unmodulated waves, amplitude-, frequency-, or phase-modulated waves and based upon the Doppler effect resulting from movement of targets
• CPC Class Codes: G01S 15/62 Sense-of-movement determination G01S15/588 takes precedence
• CPC Class Codes: G01S 15/66 Sonar tracking systems
• CPC Class Codes: G01S 15/87 Combinations of sonar systems
• CPC Class Codes: G01S 2015/465 by Trilateration, i.e. two transducers determine separately the distance to a target, whereby with the knowledge of the baseline length, i.e. the distance between the transducers, the position data of the target is determined

Abstract

A method, system and computer program product for tracking movement of an object, such as a hand. Speakers of a device to be controlled transmit frequency modulated continuous wave (FMCW) audio signals. These signals are reflected by the object and received by the microphones at the controlled device. The received and transmitted audio signals are mixed. A fast Fourier transform (FFT) is then performed on the mixed audio signals. One or more peak frequencies in the frequency domain of the FFT mixed audio signals are selected and used to estimate the distance between the object and the speakers of the controlled device. Furthermore, the velocity of the object is estimated. The coordinates of the object are then computed using the estimated distance between the object and the speakers and microphones of the controlled device and the estimated velocity of the object.

Claims

1. A method for tracking movement of an object, the method comprising:

transmitting audio signals from one or more speakers to said object;

receiving samples of said audio signals reflected from said object over a period of time;

mixing said received audio signals with said transmitted audio signals;

performing, by a processor, a fast Fourier transform on said mixed audio signals;

selecting, by said processor, one or more peak frequencies in a frequency domain of said fast Fourier transformed mixed audio signals;

estimating a velocity of said object; and

estimating, by said processor, a distance from a speaker of a controlled device to a microphone of said controlled device via said object based on said selected one or more peak frequencies and velocity of said object.

2. The method as recited in claim 1 further comprising:

estimating coordinates of said object using said estimated distance from said speaker of said controlled device to said microphone of said controlled device via said object, said estimated velocity of said object, estimated distances between all speakers of said controlled device and an estimated distance between each pair of said speakers and microphones of said controlled device.

3. The method as recited in claim 1 further comprising:

estimating a pseudo Doppler shift in said frequency domain of said fast Fourier transformed mixed audio signals, wherein said pseudo Doppler shift is a difference between a current peak frequency versus a closest spectral point below said current peak frequency in said fast Fourier transformed mixed audio signals; and

translating said pseudo Doppler shift to an actual Doppler shift to estimate said velocity of said object.

4. The method as recited in claim 3 further comprising:

detecting peak frequencies in said fast Fourier transformed mixed audio signals;

filtering peak frequencies at spectral points;

selecting peak frequencies from remaining unfiltered peak frequencies;

estimating a Doppler shift for each of the selected peak frequencies; and

combining said estimated Doppler shifts to derive said pseudo Doppler shift.

5. The method as recited in claim 3 further comprising:

translating said pseudo Doppler shift to said actual Doppler shift within a time window based on a change in said pseudo Doppler shift within said time window.

6. The method as recited in claim 3 further comprising:

selecting one or more peak frequencies in said frequency domain of said fast Fourier transformed mixed audio signals based on their magnitude and said estimated velocity of said object.

7. The method as recited in claim 1 further comprising:

estimating an initial position of said object using said estimated distance from said speaker of said controlled device to said microphone of said controlled device via said object; and

computing coordinates of said object using said estimated distance from said speaker of said controlled device to said microphone of said controlled device via said object, said estimated velocity of said object and said initial position of said object.

8. The method as recited in claim 7, wherein an initial distance of said object is estimated by filtering out spectral points, selecting a peak out of the remaining peaks based on their amplitude and/or consistency with said estimated velocity of said object estimated from an actual Doppler shift, rounding said selected peak to the closest spectral points to form a rounded frequency, and converting said rounded frequency to said initial distance estimate of said object.

9. The method as recited in claim 1 further comprising:

estimating a distance between said speaker and said microphone of said controlled device;

estimating a distance between speakers of said controlled device using said distance between said speaker and said microphone of said controlled device; and

computing coordinates of said object using said estimated distance from said speaker of said controlled device to said microphone of said controlled device via said object, said estimated velocity of said object, said estimated distance between said speakers of said controlled device and an estimated distance between said speakers and microphones of said controlled device.

10. The method as recited in claim 1 further comprising:

detecting an initial gesture; and

estimating an initial position of said object in response to detecting said initial gesture.

11. The method as recited in claim 10 further comprising:

detecting said initial gesture by collecting positive traces of said fast Fourier transformed mixed audio signals while a user is performing said initial gesture and collecting negative traces of said fast Fourier transformed mixed audio signals while said user is not performing said initial gesture; and

matching recently received fast Fourier transformed mixed audio signals against said positive or negative traces using a pattern matching algorithm to detect said initial gesture.

12. The method as recited in claim 1 further comprising:

filtering out a reflection from static objects.

13. The method as recited in claim 1, wherein said object is a hand or a pen.

14. The method as recited in claim 1, wherein said controlled device comprises one of the following:

a computer, a video game console, an augmented reality/virtual reality headset, a smartphone, a smart watch, a wearable device, a smart TV and a smart appliance.

15. A computer program product for tracking movement of an object, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code comprising the programming instructions for:

transmitting audio signals from one or more speakers to said object;

receiving samples of said audio signals reflected from said object over a period of time;

mixing said received audio signals with said transmitted audio signals;

performing a fast Fourier transform on said mixed audio signals;

selecting one or more peak frequencies in a frequency domain of said fast Fourier transformed mixed audio signals;

estimating a velocity of said object; and

estimating a distance from a speaker of a controlled device to a microphone of said controlled device via said object based on said selected one or more peak frequencies and velocity of said object.

16. The computer program product as recited in claim 15, wherein the program code further comprises the programming instructions for:

estimating coordinates of said object using said estimated distance from said speaker of said controlled device to said microphone of said controlled device via said object, said estimated velocity of said object, estimated distances between all speakers of said controlled device and an estimated distance between each pair of said speakers and microphones of said controlled device.

17. The computer program product as recited in claim 15, wherein the program code further comprises the programming instructions for:

estimating a pseudo Doppler shift in said frequency domain of said fast Fourier transformed mixed audio signals, wherein said pseudo Doppler shift is a difference between a current peak frequency versus a closest spectral point below said current peak frequency in said fast Fourier transformed mixed audio signals; and

translating said pseudo Doppler shift to an actual Doppler shift to estimate said velocity of said object.

18. The computer program product as recited in claim 17, wherein the program code further comprises the programming instructions for:

detecting peak frequencies in said fast Fourier transformed mixed audio signals;

filtering peak frequencies at spectral points;

selecting peak frequencies from remaining unfiltered peak frequencies;

estimating a Doppler shift for each of the selected peak frequencies; and

combining said estimated Doppler shifts to derive said pseudo Doppler shift.

19. The computer program product as recited in claim 17, wherein the program code further comprises the programming instructions for:

translating said pseudo Doppler shift to said actual Doppler shift within a time window based on a change in said pseudo Doppler shift within said time window.

20. The computer program product as recited in claim 17, wherein the program code further comprises the programming instructions for:

selecting one or more peak frequencies in said frequency domain of said fast Fourier transformed mixed audio signals based on their magnitude and said estimated velocity of said object.

21. The computer program product as recited in claim 15, wherein the program code further comprises the programming instructions for:

estimating an initial position of said object using said estimated distance from said speaker of said controlled device to said microphone of said controlled device via said object; and

computing coordinates of said object using said estimated distance from said speaker of said controlled device to said microphone of said controlled device via said object, said estimated velocity of said object and said initial position of said object.

22. The computer program product as recited in claim 21, wherein an initial distance of said object is estimated by filtering out spectral points, selecting a peak out of the remaining peaks based on their amplitude and/or consistency with said estimated velocity of said object estimated from an actual Doppler shift, rounding said selected peak to the closest spectral points to form a rounded frequency, and converting said rounded frequency to said initial distance estimate of said object.

23. The computer program product as recited in claim 15, wherein the program code further comprises the programming instructions for:

estimating a distance between said speaker and said microphone of said controlled device;

estimating a distance between speakers of said controlled device using said distance between said speaker and said microphone of said controlled device; and

computing coordinates of said object using said estimated distance from said speaker of said controlled device to said microphone of said controlled device via said object, said estimated velocity of said object, said estimated distance between said speakers of said controlled device and an estimated distance between said speakers and microphones of said controlled device.

24. The computer program product as recited in claim 15, wherein the program code further comprises the programming instructions for:

detecting an initial gesture; and

estimating an initial position of said object in response to detecting said initial gesture.

25. The computer program product as recited in claim 24, wherein the program code further comprises the programming instructions for:

detecting said initial gesture by collecting positive traces of said fast Fourier transformed mixed audio signals while a user is performing said initial gesture and collecting negative traces of said fast Fourier transformed mixed audio signals while said user is not performing said initial gesture; and

matching recently received fast Fourier transformed mixed audio signals against said positive or negative traces using a pattern matching algorithm to detect said initial gesture.

26. The computer program product as recited in claim 15, wherein the program code further comprises the programming instructions for:

filtering out a reflection from static objects.

27. A controlled device, comprising:

a memory for storing a computer program for tracking movement of an object; and

a processor coupled to the memory, wherein the processor is configured to execute the program instructions of the computer program comprising;

transmitting audio signals from one or more speakers to said object;

receiving samples of said audio signals reflected from said object over a period of time;

mixing said received audio signals with said transmitted audio signals;

performing a fast Fourier transform on said mixed audio signals;

selecting one or more peak frequencies in a frequency domain of said fast Fourier transformed mixed audio signals;

estimating a velocity of said object; and

estimating a distance from a speaker of said controlled device to a microphone of said controlled device via said object based on said selected one or more peak frequencies and velocity of said object.

28. The controlled device as recited in claim 27, wherein the program instructions of the computer program further comprise:

estimating coordinates of said object using said estimated distance from said speaker of said controlled device to said microphone of said controlled device via said object, said estimated velocity of said object, estimated distances between all speakers of said controlled device and an estimated distance between each pair of said speakers and microphones of said controlled device.

29. The controlled device as recited in claim 27, wherein the program instructions of the computer program further comprise:

estimating a pseudo Doppler shift in said frequency domain of said fast Fourier transformed mixed audio signals, wherein said pseudo Doppler shift is a difference between a current peak frequency versus a closest spectral point below said current peak frequency in said fast Fourier transformed mixed audio signals; and

translating said pseudo Doppler shift to an actual Doppler shift to estimate said velocity of said object.

30. The controlled device as recited in claim 29, wherein the program instructions of the computer program further comprise:

detecting peak frequencies in said fast Fourier transformed mixed audio signals;

filtering peak frequencies at spectral points;

selecting peak frequencies from remaining unfiltered peak frequencies;

estimating a pseudo Doppler shift for each of the selected peak frequencies; and

combining said estimated pseudo Doppler shifts to derive a final pseudo Doppler shift.

31. The controlled device as recited in claim 29, wherein the program instructions of the computer program further comprise:

translating said pseudo Doppler shift to said actual Doppler shift within a time window based on a change in said pseudo Doppler shift within said time window.

32. The controlled device as recited in claim 29, wherein the program instructions of the computer program further comprise:

selecting one or more peak frequencies in said frequency domain of said fast Fourier transformed mixed audio signals based on their magnitude and said estimated velocity of said object.

33. The controlled device as recited in claim 27, wherein the program instructions of the computer program further comprise:

estimating an initial position of said object using said estimated distance from said speaker of said controlled device to said microphone of said controlled device via said object; and

computing coordinates of said object using said estimated distance from said speaker of said controlled device to said microphone of said controlled device via said object, said estimated velocity of said object and said initial position of said object.

34. The controlled device as recited in claim 33, wherein an initial distance of said object is estimated by filtering out spectral points, selecting a peak out of the remaining peaks based on their amplitude and/or consistency with said estimated velocity of said object estimated from an actual Doppler shift, rounding said selected peak to the closest spectral points to form a rounded frequency, and converting said rounded frequency to said initial distance estimate of said object.

35. The controlled device as recited in claim 27, wherein the program instructions of the computer program further comprise:

estimating a distance between said speaker and said microphone of said controlled device;

estimating a distance between speakers of said controlled device using said distance between said speaker and said microphone of said controlled device; and

computing coordinates of said object using said estimated distance from said speaker of said controlled device to said microphone of said controlled device via said object, said estimated velocity of said object, said estimated distance between said speakers of said controlled device and an estimated distance between said speakers and microphones of said controlled device.

36. The controlled device as recited in claim 27, wherein the program instructions of the computer program further comprise:

detecting an initial gesture; and

estimating an initial position of said object in response to detecting said initial gesture.

37. The controlled device as recited in claim 36, wherein the program instructions of the computer program further comprise:

detecting said initial gesture by collecting positive traces of said fast Fourier transformed mixed audio signals while a user is performing said initial gesture and collecting negative traces of said fast Fourier transformed mixed audio signals while said user is not performing said initial gesture; and

matching recently received fast Fourier transformed mixed audio signals against said positive or negative traces using a pattern matching algorithm to detect said initial gesture.

38. The controlled device as recited in claim 27, wherein the program instructions of the computer program further comprise:

filtering out a reflection from static objects.