TOUCHLESS SENSING AND GESTURE RECOGNITION USING CONTINUOUS WAVE ULTRASOUND SIGNALS

• US 20120001875A1
• Filed: 09/17/2010
• Published: 01/05/2012
• Est. Priority Date: 06/29/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Li, Ren, Lee, Te-Won, Gupta, Samir K., Nelson, Hui-ya L.
• Granted Patent: US 8,907,929 B2
• Time in Patent Office: Days
• US Class Current: 345/177
• CPC Class Codes: G01S 15/325 using transmission of coded signals, e.g. of phase-shift keyed [PSK] signals
• CPC Class Codes: G01S 15/50 Systems of measurement, based on relative movement of the target
• CPC Class Codes: G01S 15/876 Combination of several spaced transmitters or receivers of known location for determining the position of a transponder or a reflector G01S15/872 takes precedence
• CPC Class Codes: G01S 7/5273 using digital techniques
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

The embodiments provide systems and methods for touchless sensing and gesture recognition using continuous wave sound signals. Continuous wave sound, such as ultrasound, emitted by a transmitter may reflect from an object, and be received by one or more sound receivers. Sound signals may be temporally encoded. Received sound signals may be processed to determine a channel impulse response or calculate time of flight. Determined channel impulse responses may be processed to extract recognizable features or angles. Extracted features may be compared to a database of features to identify a user input gesture associated with the matched feature. Angles of channel impulse response curves may be associated with an input gesture. Time of flight values from each receiver may be used to determine coordinates of the reflecting object. Embodiments may be implemented as part of a graphical user interface. Embodiments may be used to determine a location of an emitter.

Claims

1. A method for recognizing a user input gesture, comprising:

receiving in a sound detector temporally encoded ultrasound signals;

calculating a channel impulse response based on the received signals;

processing the channel impulse response to extract features;

comparing the extracted feature to a database of channel impulse response features to identify a matching store feature; and

executing a command associated with the matched stored feature.

2. The method of claim 1, further comprising transmitting the temporally encoded continuous wave sound signal from an ultrasound emitter.

3. The method of claim 1, wherein the sound is ultrasound.

4. The method of claim 2, wherein the sound is temporally encoded with pseudorandom noise.

5. The method of claim 2, wherein the sound is temporally encoded with a changing frequency.

6. The method of claim 2, wherein the sound is temporally encoded using a spread spectrum modulation.

7. The method of claim 1, wherein receiving sound signals in a sound detector comprises receiving sound signals in three sound detectors spaced apart on a computing device.

8. The method of claim 1, wherein processing the channel impulse response to extract features comprises subtracting a background channel impulse response to obtain a reflector channel impulse response.

9. The method of claim 8, wherein processing the channel impulse response to extract features further comprises calculating a difference channel impulse response and processing a sequence of different channel impulse responses as an image.

10. The method of claim 9, wherein processing the channel impulse response to extract features further comprises applying an edge filter to the difference channel impulse response image.

11. The method of claim 9, wherein processing the channel impulse response to extract features further comprises applying a grey scale correlation matrix to the difference channel impulse response image.

12. The method of claim 9, wherein comparing the extracted feature to a database the channel impulse response features comprises performing an analysis using a method selected from a group consisting of k-nearest neighbors analysis, artificial neural nets analysis, and hidden Markov models analysis.

13. The method of claim 1, further comprising:

identifying a strongest reflector based on the channel impulse response features;

measuring a time of flight of sound from the identify strongest reflector; and

determining a location in space of the identified strongest reflector based upon the measured time of flight.

14. The method of claim 13, further comprising applying Kalman filtering to the measured time of flight data.

15. The method of claim 1, further comprising determining a channel impulse response curve based on the calculate channel impulse response,

wherein processing the channel impulse response to extract features comprises determining a reflector movement direction based upon angle information derived from the channel impulse response curve.

16. A method for recognizing a user input gesture, comprising:

receiving in a sound detector temporally encoded ultrasound signals;

calculating a channel impulse response based on the received signals;

processing the channel impulse response to determine an angle of a channel impulse response curve; and

executing a command associated with the determined angle of the channel impulse response curve.

17. A computer system, comprising:

a processor;

a memory coupled to the processor;

a sound emitter coupled to the processor; and

one or more sound detectors coupled to the processor,

wherein the processor is configured with processor-executable instructions to perform operations comprising;

receiving in a sound detector temporally encoded ultrasound signals;

calculating a channel response based on the received signals;

processing the channel impulse response to extract features; and

comparing the extracted feature to a database of channel impulse response features to identify a matching store feature; and

executing a command associated with the matched stored feature.

18. The computer system of claim 17, wherein the processor is configured with processor-executable instructions to perform operations further comprising transmitting the temporally encoded continuous wave sound signal from an ultrasound emitter.

19. The computer system of claim 17, wherein the processor is configured with processor-executable instructions to perform operations such that the sound is ultrasound.

20. The computer system of claim 18, wherein the processor is configured with processor-executable instructions to perform operations such that the sound is temporally encoded with pseudorandom noise.

21. The computer system of claim 18, wherein the processor is configured with processor-executable instructions to perform operations such that the sound is temporally encoded with a changing frequency.

22. The computer system of claim 18, wherein the processor is configured with processor-executable instructions to perform operations such that the sound is temporally encoded using a spread spectrum modulation.

23. The computer system of claim 17, wherein the processor is configured with processor-executable instructions to perform operations such that receiving sound signals in a sound detector comprises receiving sound signals in three sound detectors spaced apart on a computing device.

24. The computer system of claim 17, wherein the processor is configured with processor-executable instructions to perform operations such that processing the channel impulse response to extract features comprises subtracting a background channel impulse response to obtain a reflector channel impulse response.

25. The computer system of claim 24, wherein the processor is configured with processor-executable instructions to perform operations such that processing the channel impulse response to extract features further comprises calculating a difference channel impulse response and processing a sequence of different channel impulse responses as an image.

26. The computer system of claim 25, wherein the processor is configured with processor-executable instructions to perform operations such that processing the channel impulse response to extract features further comprises applying an edge filter to the difference channel impulse response image.

27. The computer system of claim 25, wherein the processor is configured with processor-executable instructions to perform operations such that processing the channel impulse response to extract features further comprises applying a grey scale correlation matrix to the difference channel impulse response image.

28. The computer system of claim 25, wherein the processor is configured with processor-executable instructions to perform operations such that comparing the extracted feature to a database the channel impulse response features comprises performing an analysis using a method selected from a group consisting of k-nearest neighbors analysis, artificial neural nets analysis, and hidden Markov models analysis.

29. The computer system of claim 17, wherein the processor is configured with processor-executable instructions to perform operations further comprising:

identifying a strongest reflector based on the channel impulse response features;

measuring a time of flight of sound from the identify strongest reflector; and

determining a location in space of the identified strongest reflector based upon the measured time of flight.

30. The computer system of claim 29, wherein the processor is configured with processor-executable instructions to perform operations further comprising applying Kalman filtering to the measured time of flight data.

31. The computer system of claim 17, wherein the processor is configured with processor-executable instructions to perform operations further comprising determining a channel impulse response curve based on the calculate channel impulse response,

wherein the processor is configured with processor-executable instructions to perform operations such that processing the channel impulse response to extract features comprises determining a reflector movement direction based upon angle information derived from the channel impulse response curve.

32. A computer system, comprising:

a processor;

a memory coupled to the processor;

a sound emitter coupled to the processor; and

one or more sound detectors coupled to the processor,

wherein the processor is configured with processor-executable instructions to perform operations comprising;

receiving in a sound detector temporally encoded ultrasound signals;

calculating a channel impulse response based on the received signals;

processing the channel impulse response to determine an angle of a channel impulse response curve; and

executing a command associated with the determined angle of the channel impulse response curve.

33. A computer system, comprising:

means for receiving in a sound detector temporally encoded ultrasound signals;

means for calculating a channel response based on the received signals;

means for processing the channel impulse response to extract features; and

means for comparing the extracted feature to a database of channel impulse response features to identify a matching store feature; and

means for executing a command associated with the matched stored feature.

34. The computer system of claim 33, further comprising means for transmitting the temporally encoded continuous wave sound signal from an ultrasound emitter.

35. The computer system of claim 33, wherein the sound is ultrasound.

36. The computer system of claim 34, wherein the sound is temporally encoded with pseudorandom noise.

37. The computer system of claim 34, wherein the sound is temporally encoded with a changing frequency.

38. The computer system of claim 34, wherein the sound is temporally encoded using a spread spectrum modulation.

39. The computer system of claim 33, wherein means for receiving sound signals in a sound detector comprises means for receiving sound signals in three sound detectors spaced apart on a computing device.

40. The computer system of claim 33, wherein means for processing the channel impulse response to extract features comprises means for subtracting a background channel impulse response to obtain a reflector channel impulse response.

41. The computer system of claim 40, wherein means for processing the channel impulse response to extract features further comprises means for calculating a difference channel impulse response and processing a sequence of different channel impulse responses as an image.

42. The computer system of claim 41, wherein means for processing the channel impulse response to extract features further comprises means for applying an edge filter to the difference channel impulse response image.

43. The computer system of claim 41, wherein means for processing the channel impulse response to extract features further comprises means for applying a grey scale correlation matrix to the difference channel impulse response image.

44. The computer system of claim 41, wherein means for comparing the extracted feature to a database the channel impulse response features comprises means for performing an analysis using a method selected from a group consisting of k-nearest neighbors analysis, artificial neural nets analysis, and hidden Markov models analysis.

45. The computer system of claim 33, further comprising:

means for identifying a strongest reflector based on the channel impulse response features;

means for measuring a time of flight of sound from the identify strongest reflector; and

means for determining a location in space of the identified strongest reflector based upon the measured time of flight.

46. The computer system of claim 45, further comprising means for applying Kalman filtering to the measured time of flight data.

47. The computer system of claim 33, further comprising means for determining a channel impulse response curve based on the calculate channel impulse response,

wherein means for processing the channel impulse response to extract features comprises means for determining a reflector movement direction based upon angle information derived from the channel impulse response curve.

48. A computer system, comprising:

means for receiving in a sound detector temporally encoded ultrasound signals;

means for calculating a channel impulse response based on the received signals;

means for processing the channel impulse response to determine an angle of a channel impulse response curve; and

means for executing a command associated with the determined angle of the channel impulse response curve.

49. A non-transitory computer readable storage medium having stored thereon computer-executable instructions configured to cause a computer configured to emit and detect ultrasound to perform operations comprising:

receiving in a sound detector temporally encoded ultrasound signals;

calculating a channel response based on the received signals;

processing the channel impulse response to extract features; and

comparing the extracted feature to a database of channel impulse response features to identify a matching store feature; and

executing a command associated with the matched stored feature.

50. The non-transitory computer readable storage medium of claim 49, wherein the stored computer-executable instructions are configured to cause a computer to perform operations further comprising transmitting the temporally encoded continuous wave sound signal from an ultrasound emitter.

51. The non-transitory computer readable storage medium of claim 49, wherein the stored computer-executable instructions are configured to cause a computer to perform operations such that the sound is ultrasound.

52. The non-transitory computer readable storage medium of claim 50, wherein the stored computer-executable instructions are configured to cause a computer to perform operations such that the sound is temporally encoded with pseudorandom noise.

53. The non-transitory computer readable storage medium of claim 50, wherein the stored computer-executable instructions are configured to cause a computer to perform operations such that the sound is temporally encoded with a changing frequency.

54. The non-transitory computer readable storage medium of claim 50, wherein the stored computer-executable instructions are configured to cause a computer to perform operations such that the sound is temporally encoded using a spread spectrum modulation.

55. The non-transitory computer readable storage medium of claim 49, wherein the stored computer-executable instructions are configured to cause a computer to perform operations such that receiving sound signals in a sound detector comprises receiving sound signals in three sound detectors spaced apart on a computing device.

56. The non-transitory computer readable storage medium of claim 49, wherein the stored computer-executable instructions are configured to cause a computer to perform operations such that processing the channel impulse response to extract features comprises subtracting a background channel impulse response to obtain a reflector channel impulse response.

57. The non-transitory computer readable storage medium of claim 56, wherein the stored computer-executable instructions are configured to cause a computer to perform operations such that processing the channel impulse response to extract features further comprises calculating a difference channel impulse response and processing a sequence of different channel impulse responses as an image.

58. The non-transitory computer readable storage medium of claim 57, wherein the stored computer-executable instructions are configured to cause a computer to perform operations such that processing the channel impulse response to extract features further comprises applying an edge filter to the difference channel impulse response image.

59. The non-transitory computer readable storage medium of claim 57, wherein the stored computer-executable instructions are configured to cause a computer to perform operations such that processing the channel impulse response to extract features further comprises applying a grey scale correlation matrix to the difference channel impulse response image.

60. The non-transitory computer readable storage medium of claim 57, wherein the stored computer-executable instructions are configured to cause a computer to perform operations such that comparing the extracted feature to a database the channel impulse response features comprises performing an analysis using a method selected from a group consisting of k-nearest neighbors analysis, artificial neural nets analysis, and hidden Markov models analysis.

61. The non-transitory computer readable storage medium of claim 49, wherein the stored computer-executable instructions are configured to cause a computer to perform operations further comprising:

identifying a strongest reflector based on the channel impulse response features;

measuring a time of flight of sound from the identify strongest reflector; and

determining a location in space of the identified strongest reflector based upon the measured time of flight.

62. The non-transitory computer readable storage medium of claim 61, wherein the stored computer-executable instructions are configured to cause a computer to perform operations further comprising applying Kalman filtering to the measured time of flight data.

63. The non-transitory computer readable storage medium of claim 49, wherein the stored computer-executable instructions are configured to cause a computer to perform operations further comprising determining a channel impulse response curve based on the calculate channel impulse response,

wherein the stored computer-executable instructions are configured to cause a computer to perform operations such that processing the channel impulse response to extract features comprises determining a reflector movement direction based upon angle information derived from the channel impulse response curve.

64. A non-transitory computer readable storage medium having stored thereon computer-executable instructions configured to cause a computer configured to emit and detect ultrasound to perform operations comprising:

receiving in a sound detector temporally encoded ultrasound signals;

calculating a channel impulse response based on the received signals;

processing the channel impulse response to determine an angle of a channel impulse response curve; and

executing a command associated with the determined angle of the channel impulse response curve.

65. A method for determining relative coordinates between a first device and a second device, comprising:

transmitting a temporally encoded continuous wave ultrasound signal from an ultrasound emitter of the first device;

receiving the temporally encoded continuous wave ultrasound signal at the second device;

measuring a time of flight of the received temporally encoded continuous wave ultrasound signal;

applying Kalman filtering to the measured time of flight; and

determining a location of the first device relative to the second device based upon the measured time of flight.

66. The method of claim 65, further comprising:

converting coordinates into a distance.

67. A computer system, comprising:

a processor;

a memory coupled to the processor;

a sound emitter coupled to the processor; and

one or more sound detectors coupled to the processor,

wherein the processor is configured with processor-executable instructions to perform operations comprising;

transmitting a temporally encoded continuous wave ultrasound signal from an ultrasound emitter of the first device;

receiving the temporally encoded continuous wave ultrasound signal at the second device;

measuring a time of flight of the received temporally encoded continuous wave ultrasound signal;

applying Kalman filtering to the measured time of flight; and

determining a location of the first device relative to the second device based upon the measured time of flight.

68. A computer system, comprising:

means for transmitting a temporally encoded continuous wave ultrasound signal from an ultrasound emitter of the first device;

means for receiving the temporally encoded continuous wave ultrasound signal at the second device;

means for measuring a time of flight of the received temporally encoded continuous wave ultrasound signal;

means for applying Kalman filtering to the measured time of flight; and

means for determining a location of the first device relative to the second device based upon the measured time of flight.

69. A non-transitory computer readable storage medium having stored thereon computer-executable instructions configured to cause a computer configured to emit and detect ultrasound to perform operations comprising:

transmitting a temporally encoded continuous wave ultrasound signal from an ultrasound emitter of the first device;

receiving the temporally encoded continuous wave ultrasound signal at the second device;

measuring a time of flight of the received temporally encoded continuous wave ultrasound signal;

applying Kalman filtering to the measured time of flight; and

determining a location of the first device relative to the second device based upon the measured time of flight.