Detecting, classifying, and interpreting input events based on stimuli in multiple sensory domains

• US 20030132950A1
• Filed: 06/28/2002
• Published: 07/17/2003
• Est. Priority Date: 11/27/2001
• Status: Abandoned Application

• Current Assignee: Canesta Incorporated (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Surucu, Fahri, Tomasi, Carlo
• Application Number:
  US10/187,032
• Publication Number:
  US 20030132950A1
• Time in Patent Office: Days
• US Class Current: 345/700
• CPC Class Codes: G01S 17/06 Systems determining position data of a target
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1632 External expansion units, e.g. docking stations
• CPC Class Codes: G06F 1/166 related to integrated arrangements for adjusting the position of the main body with respect to the supporting surface, e.g. legs for adjusting the tilt angle
• CPC Class Codes: G06F 1/1673 Arrangements for projecting a virtual keyboard
• CPC Class Codes: G06F 2200/1633 Protecting arrangement for the entire housing of the computer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

Stimuli in two or more sensory domains, such as an auditory domain and a visual domain, are combined in order to improve reliability and accuracy of detected user input. Detected events that occur substantially simultaneously in the multiple domains are deemed to represent the same user action, and if interpretable as a coherent action and provided to the system as interpreted input. The invention is applicable, for example, in a virtual keyboard or virtual controller, where stimuli resulting from user actions are detected, interpreted, and provided as input to a system.

Claims

1. A computer-implemented method for classifying an input event, the method comprising:

receiving, at a visual sensor, a first stimulus resulting from user action, in a visual domain;

receiving, at an auditory sensor, a second stimulus resulting from user action, in an auditory domain; and

responsive to the first and second stimuli indicating substantial simultaneity of the corresponding user action, classifying the stimuli as associated with a single user input event.

5. The method of claim 1 or 2, further comprising:

generating a vector of visual features based on the first stimulus;

generating a vector of acoustic features based on the second stimulus;

comparing the generated vectors to user action descriptors for a plurality of user actions; and

responsive to the comparison indicating a match, outputting a signal indicating a recognized user action.

6. The method of claim 1 or 2, wherein the single user input event comprises a keystroke.

7. The method of claim 1 or 2, wherein each user action comprises a physical gesture.

8. The method of claim 1 or 2, wherein each user action comprises at least one virtual key press.

9. The method of claim 1 or 2, wherein receiving a first stimulus comprises receiving a stimulus at a camera.

10. The method of claim 1 or 2, wherein receiving a second stimulus comprises receiving a stimulus at a microphone.

11. The method of claim 1 or 2, further comprising:

determining a series of waveform signals from the received second stimulus; and

comparing the waveform signals to at least one predetermined waveform sample to determine occurrence and time of at least one auditory event.

12. The method of claim 1 or 2, further comprising:

determining a series of sound intensity values from the received second stimulus; and

comparing the sound intensity values with at a threshold value to determine occurrence and time of at least one auditory event.

13. The method of claim 1 or 2, wherein receiving a second stimulus comprises receiving an acoustic stimulus representing a user'"'"'s taps on a surface.

14. The method of claim 1 or 2, further comprising:

responsive to the stimuli being classified as associated with a single user input event, transmitting a command associated with the user input event.

15. The method of claim 1 or 2, further comprising:

determining a metric measuring relative force of the user action; and

generating a parameter for the user input event based on the determined force metric.

16. The method of claim 1 or 2, further comprising transmitting the classified input event to one selected from the group consisting of:

a computer;

a handheld computer;

a personal digital assistant;

a musical instrument; and

a remote control.

17. The method of claim 1, further comprising:

for each received stimulus, determining a probability that the stimulus represents an intended user action; and

combining the determined probabilities to determine an overall probability that the received stimuli collectively represent a single intended user action.

18. The method of claim 1, further comprising:

for each received stimulus, determining a time for the corresponding user action; and

comparing the determined time to determine whether the first and second stimuli indicate substantial simultaneity of the corresponding user action.

19. The method of claim 1, further comprising:

for each received stimulus, reading a time stamp indicating a time for the corresponding user action; and

comparing the time stamps to determine whether the first and second stimuli indicate substantial simultaneity of the corresponding user action.

2. A computer-implemented method for classifying an input event, comprising:

receiving a first stimulus, resulting from user action, in a visual domain;

receiving a second stimulus, resulting from user action, in an auditory domain;

classifying the first stimulus according to at least a time of occurrence;

classifying the second stimulus according to at least a time of occurrence; and

responsive to the classifying steps indicating substantial simultaneity of the first and second stimuli, classifying the stimuli as associated with a single user input event.

3. The method of claim 2, wherein:

classifying the first stimulus comprises determining a time for the corresponding user action; and

classifying the second stimulus comprises determining a time for the corresponding user action.

4. The method of claim 3, wherein:

determining a time comprises reading a time stamp.

20. A computer-implemented method for filtering input events, comprising:

detecting, in a visual domain, a first plurality of input events resulting from user action;

detecting, in an auditory domain, a second plurality of input events resulting from user action;

for each detected event in the first plurality;

determining whether the detected event in the first plurality corresponds to a detected event in the second plurality; and

responsive to the detected event in the first plurality not corresponding to a detected event in the second plurality, filtering out the event in the first plurality.

21. The method of claim 20, wherein determining whether the detected event in the first plurality corresponds to a detected event in the second plurality comprises:

determining whether the detected event in the first plurality and the detected event in the second plurality occurred substantially simultaneously.

22. The method of claim 20, wherein determining whether the detected event in the first plurality corresponds to a detected event in the second plurality comprises:

determining whether the detected event in the first plurality and the detected event in the second plurality respectively indicate substantially simultaneous user actions.

23. The method of claim 20, wherein each user action comprises at least one physical gesture.

24. The method of claim 20, wherein each user action comprises at least one virtual key press.

25. The method of claim 20, wherein detecting a first plurality of input events comprises receiving signals from a camera.

26. The method of claim 20, wherein detecting a second plurality of input events comprises receiving signals from a microphone.

27. The method of claim 20, further comprising, for each detected event in the first plurality:

responsive to the event not being filtered out, transmitting a command associated with the event.

28. The method of claim 27, further comprising, responsive to the event not being filtered out:

determining a metric measuring relative force of the user action; and

generating a parameter for the command based on the determined force metric.

29. The method of claim 20, wherein determining whether the detected event in the first plurality corresponds to a detected event in the second plurality comprises:

determining whether a time stamp for the detected event in the first plurality indicates substantially the same time as a time stamp for the detected event in the second plurality.

30. A computer-implemented method for classifying an input event, comprising:

receiving a visual stimulus, resulting from user action, in a visual domain;

receiving an acoustic stimulus, resulting from user action, in an auditory domain; and

generating a vector of visual features based on the received visual stimulus;

generating a vector of acoustic features based on the received acoustic stimulus;

comparing the generated vectors to user action descriptors for a plurality of user actions; and

responsive to the comparison indicating a match, outputting a signal indicating a recognized user action.

31. A system for classifying an input event, comprising:

an optical sensor, for receiving an optical stimulus resulting from user action, in a visual domain, and for generating a first signal representing the optical stimulus;

an acoustic sensor, for receiving an acoustic stimulus resulting from user action, in an auditory domain, and for generating a second signal representing the acoustic stimulus; and

a synchronizer, coupled to receive the first signal from the optical sensor and the second signal from the acoustic sensor, for determining whether the received signals indicate substantial simultaneity of the corresponding user action, and responsive to the determination, classifying the signals as associated with a single user input event.

32. The system of claim 31, wherein the user action comprises at least one keystroke.

33. The system of claim 31, wherein the user action comprises at least one physical gesture.

34. The system of claim 31, further comprising:

a virtual keyboard, positioned to guide user actions to result in stimuli detectable by the optical and acoustic sensors;

wherein a user action comprises a key press on the virtual keyboard.

35. The system of claim 31, wherein the optical sensor comprises a camera.

36. The system of claim 31, wherein the acoustic sensor comprises a transducer.

37. The system of claim 31, wherein the acoustic sensor generates at least one waveform signal representing the second stimulus, the system further comprising:

a processor, coupled to the synchronizer, for comparing the at least one waveform signal with at least one predetermined waveform sample to determining occurrence and time of at least one auditory event.

38. The system of claim 31, wherein the acoustic sensor generates at least one waveform intensity value representing the second stimulus, the system further comprising:

a processor, coupled to the synchronizer, for comparing the at least one waveform intensity value with at least one predetermined threshold value to determining occurrence and time of at least one auditory event.

39. The system of claim 31, further comprising:

a surface for receiving a user'"'"'s taps;

wherein the acoustic sensor receives an acoustic stimulus representing the user'"'"'s taps on the surface.

40. The system of claim 31, further comprising:

a processor, coupled to the synchronizer, for, responsive to the stimuli being classified as associated with a single user input event, transmitting a command associated with the user input event.

41. The system of claim 31, wherein the processor:

determines a metric measuring relative force of the user action; and

generates a parameter for the command based on the determined force metric.

42. The system of claim 31, further comprising:

a processor, coupled to the synchronizer, for;

for each received stimulus, determining a probability that the stimulus represents an intended user action; and

combining the determined probabilities to determine an overall probability that the received stimuli collectively represent an intended user action.

43. The system of claim 31, wherein the synchronizer:

for each received stimulus, determines a time for the corresponding user action; and

compares the determined time to determine whether the optical and acoustic stimuli indicate substantial simultaneity of the corresponding user action.

44. The system of claim 31, wherein the synchronizer:

for each received stimulus, reads a time stamp indicating a time for the corresponding user action; and

compares the read time stamps to determine whether the optical and acoustic stimuli indicate substantial simultaneity of the corresponding user action.

45. The system of claim 31, further comprising:

a processor, coupled to the synchronizer, for identifying an intended user action, the processor comprising;

a visual feature computation module, for generating a vector of visual features based on the received optical stimulus;

an acoustic feature computation module, for generating a vector of acoustic features based on the received acoustic stimulus;

an action list containing descriptors of a plurality of user actions; and

a recognition function, coupled to the feature computation modules and to the action list, for comparing the generated vectors to the user action descriptors.

46. The system of claim 31, wherein the user input event corresponds to input for a device selected from the group consisting of:

a computer;

a handheld computer;

a personal digital assistant;

a musical instrument; and

a remote control.

47. A computer program product for classifying an input event, the computer program product comprising:

a computer readable medium; and

computer program instructions, encoded on the medium, for controlling a processor to perform the operations of;

receiving, at a visual sensor, a first stimulus resulting from user action, in a visual domain;

receiving, at an auditory sensor, a second stimulus resulting from user action, in an auditory domain; and

responsive to the first and second stimuli indicating substantial simultaneity of the corresponding user action, classifying the stimuli as associated with a single user input event.

51. The computer program product of claim 47 or 48, further comprising computer program instructions, encoded on the medium, for controlling a processor to perform the operations of:

generating a vector of visual features based on the first stimulus;

generating a vector of acoustic features based on the second stimulus;

comparing the generated vectors to user action descriptors for a plurality of user actions; and

responsive to the comparison indicating a match, outputting a signal indicating a recognized user action.

52. The computer program product of claim 47 or 48, wherein the single user input event comprises a keystroke.

53. The computer program product of claim 47 or 48, wherein each user action comprises a physical gesture.

54. The computer program product of claim 47 or 48, wherein each user action comprises at least one virtual key press.

55. The computer program product of claim 47 or 48, wherein receiving a first stimulus comprises receiving a stimulus at a camera.

56. The computer program product of claim 47 or 48, wherein receiving a second stimulus comprises receiving a stimulus at a microphone.

57. The computer program product of claim 47 or 48, further comprising computer program instructions, encoded on the medium, for controlling a processor to perform the operations of:

determining a series of waveform signals from the received second stimulus; and

comparing the waveform signals to at least one predetermined waveform sample to determine occurrence and time of at least one auditory event.

58. The computer program product of claim 47 or 48, further comprising computer program instructions, encoded on the medium, for controlling a processor to perform the operations of:

determining a series of sound intensity values from the received second stimulus; and

comparing the sound intensity values with at a threshold value to determine occurrence and time of at least one auditory event.

59. The computer program product of claim 47 or 48, wherein receiving a second stimulus comprises receiving an acoustic stimulus representing a user'"'"'s taps on a surface.

60. The computer program product of claim 47 or 48, further comprising computer program instructions, encoded on the medium, for controlling a processor to perform the operation of:

responsive to the stimuli being classified as associated with a single user input event, transmitting a command associated with the user input event.

61. The computer program product of claim 47 or 48, further comprising computer program instructions, encoded on the medium, for controlling a processor to perform the operations of:

determining a metric measuring relative force of the user action; and

generating a parameter for the user input event based on the determined force metric.

62. The computer program product of claim 47 or 48, further comprising computer program instructions, encoded on the medium, for controlling a processor to perform the operation of transmitting the classified input event to one selected from the group consisting of:

a computer;

a handheld computer;

a personal digital assistant;

a musical instrument; and

a remote control.

63. The computer program product of claim 47, further comprising computer program instructions, encoded on the medium, for controlling a processor to perform the operations of:

for each received stimulus, determining a probability that the stimulus represents an intended user action; and

combining the determined probabilities to determine an overall probability that the received stimuli collectively represent a single intended user action.

64. The computer program product of claim 47, further comprising computer program instructions, encoded on the medium, for controlling a processor to perform the operations of:

for each received stimulus, determining a time for the corresponding user action; and

comparing the determined time to determine whether the first and second stimuli indicate substantial simultaneity of the corresponding user action.

65. The computer program product of claim 47, further comprising computer program instructions, encoded on the medium, for controlling a processor to perform the operations of:

for each received stimulus, reading a time stamp indicating a time for the corresponding user action; and

comparing the time stamps to determine whether the first and second stimuli indicate substantial simultaneity of the corresponding user action.

48. A computer program product for classifying an input event, the computer program product comprising:

a computer readable medium; and

computer program instructions, encoded on the medium, for controlling a processor to perform the operations of;

receiving a first stimulus, resulting from user action, in a visual domain;

receiving a second stimulus, resulting from user action, in an auditory domain;

classifying the first stimulus according to at least a time of occurrence;

classifying the second stimulus according to at least a time of occurrence; and

responsive to the classifying steps indicating substantial simultaneity of the first and second stimuli, classifying the stimuli as associated with a single user input event.

49. The computer program product of claim 48, wherein:

classifying the first stimulus comprises determining a time for the corresponding user action; and

classifying the second stimulus comprises determining a time for the corresponding user action.

50. The computer program product of claim 49, wherein:

determining a time comprises reading a time stamp.

66. A computer program product for filtering input events, the computer program product comprising:

a computer readable medium; and

computer program instructions, encoded on the medium, for controlling a processor to perform the operations of;

detecting, in a visual domain, a first plurality of input events resulting from user action;

detecting, in an auditory domain, a second plurality of input events resulting from user action;

for each detected event in the first plurality;

determining whether the detected event in the first plurality corresponds to a detected event in the second plurality; and

responsive to the detected event in the first plurality not corresponding to a detected event in the second plurality, filtering out the event in the first plurality.

67. The computer program product of claim 66, wherein determining whether the detected event in the first plurality corresponds to a detected event in the second plurality comprises:

determining whether the detected event in the first plurality and the detected event in the second plurality occurred substantially simultaneously.

68. The computer program product of claim 66, wherein determining whether the detected event in the first plurality corresponds to a detected event in the second plurality comprises:

determining whether the detected event in the first plurality and the detected event in the second plurality respectively indicate substantially simultaneous user actions.

69. The computer program product of claim 66, wherein each user action comprises at least one physical gesture.

70. The computer program product of claim 66, wherein each user action comprises at least one virtual key press.

71. The computer program product of claim 66, wherein detecting a first plurality of input events comprises receiving signals from a camera.

72. The computer program product of claim 66, wherein detecting a second plurality of input events comprises receiving signals from a microphone.

73. The computer program product of claim 66, further comprising computer program instructions, encoded on the medium, for controlling a processor to perform the operation of, for each detected event in the first plurality:

responsive to the event not being filtered out, transmitting a command associated with the event.

74. The computer program product of claim 73, further comprising computer program instructions, encoded on the medium, for controlling a processor to perform the operations of, responsive to the event not being filtered out:

determining a metric measuring relative force of the user action; and

generating a parameter for the command based on the determined force metric.

75. The computer program product of claim 66, wherein determining whether the detected event in the first plurality corresponds to a detected event in the second plurality comprises:

determining whether a time stamp for the detected event in the first plurality indicates substantially the same time as a time stamp for the detected event in the second plurality.

76. A computer program product for classifying an input event, the computer program product comprising:

a computer readable medium; and

computer program instructions, encoded on the medium, for controlling a processor to perform the operations of;

receiving a visual stimulus, resulting from user action, in a visual domain;

receiving an acoustic stimulus, resulting from user action, in an auditory domain; and

generating a vector of visual features based on the received visual stimulus;

generating a vector of acoustic features based on the received acoustic stimulus;

comparing the generated vectors to user action descriptors for a plurality of user actions; and

responsive to the comparison indicating a match, outputting a signal indicating a recognized user action.