Detecting, classifying, and interpreting input events based on stimuli in multiple sensory domains
First Claim
Patent Images
1. A computer-implemented method for classifying an input event, the method comprising:
- receiving, at a visual sensor, a first stimulus resulting from user action, in a visual domain;
receiving, at an auditory sensor, a second stimulus resulting from user action, in an auditory domain; and
responsive to the first and second stimuli indicating substantial simultaneity of the corresponding user action, classifying the stimuli as associated with a single user input event.
1 Assignment
0 Petitions
Accused Products
Abstract
Stimuli in two or more sensory domains, such as an auditory domain and a visual domain, are combined in order to improve reliability and accuracy of detected user input. Detected events that occur substantially simultaneously in the multiple domains are deemed to represent the same user action, and if interpretable as a coherent action and provided to the system as interpreted input. The invention is applicable, for example, in a virtual keyboard or virtual controller, where stimuli resulting from user actions are detected, interpreted, and provided as input to a system.
-
Citations
76 Claims
-
1. A computer-implemented method for classifying an input event, the method comprising:
-
receiving, at a visual sensor, a first stimulus resulting from user action, in a visual domain;
receiving, at an auditory sensor, a second stimulus resulting from user action, in an auditory domain; and
responsive to the first and second stimuli indicating substantial simultaneity of the corresponding user action, classifying the stimuli as associated with a single user input event. - View Dependent Claims (5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19)
-
-
2. A computer-implemented method for classifying an input event, comprising:
-
receiving a first stimulus, resulting from user action, in a visual domain;
receiving a second stimulus, resulting from user action, in an auditory domain;
classifying the first stimulus according to at least a time of occurrence;
classifying the second stimulus according to at least a time of occurrence; and
responsive to the classifying steps indicating substantial simultaneity of the first and second stimuli, classifying the stimuli as associated with a single user input event. - View Dependent Claims (3, 4)
-
-
20. A computer-implemented method for filtering input events, comprising:
-
detecting, in a visual domain, a first plurality of input events resulting from user action;
detecting, in an auditory domain, a second plurality of input events resulting from user action;
for each detected event in the first plurality;
determining whether the detected event in the first plurality corresponds to a detected event in the second plurality; and
responsive to the detected event in the first plurality not corresponding to a detected event in the second plurality, filtering out the event in the first plurality. - View Dependent Claims (21, 22, 23, 24, 25, 26, 27, 28, 29)
-
-
30. A computer-implemented method for classifying an input event, comprising:
-
receiving a visual stimulus, resulting from user action, in a visual domain;
receiving an acoustic stimulus, resulting from user action, in an auditory domain; and
generating a vector of visual features based on the received visual stimulus;
generating a vector of acoustic features based on the received acoustic stimulus;
comparing the generated vectors to user action descriptors for a plurality of user actions; and
responsive to the comparison indicating a match, outputting a signal indicating a recognized user action.
-
-
31. A system for classifying an input event, comprising:
-
an optical sensor, for receiving an optical stimulus resulting from user action, in a visual domain, and for generating a first signal representing the optical stimulus;
an acoustic sensor, for receiving an acoustic stimulus resulting from user action, in an auditory domain, and for generating a second signal representing the acoustic stimulus; and
a synchronizer, coupled to receive the first signal from the optical sensor and the second signal from the acoustic sensor, for determining whether the received signals indicate substantial simultaneity of the corresponding user action, and responsive to the determination, classifying the signals as associated with a single user input event. - View Dependent Claims (32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46)
-
-
47. A computer program product for classifying an input event, the computer program product comprising:
-
a computer readable medium; and
computer program instructions, encoded on the medium, for controlling a processor to perform the operations of;
receiving, at a visual sensor, a first stimulus resulting from user action, in a visual domain;
receiving, at an auditory sensor, a second stimulus resulting from user action, in an auditory domain; and
responsive to the first and second stimuli indicating substantial simultaneity of the corresponding user action, classifying the stimuli as associated with a single user input event. - View Dependent Claims (51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65)
-
-
48. A computer program product for classifying an input event, the computer program product comprising:
-
a computer readable medium; and
computer program instructions, encoded on the medium, for controlling a processor to perform the operations of;
receiving a first stimulus, resulting from user action, in a visual domain;
receiving a second stimulus, resulting from user action, in an auditory domain;
classifying the first stimulus according to at least a time of occurrence;
classifying the second stimulus according to at least a time of occurrence; and
responsive to the classifying steps indicating substantial simultaneity of the first and second stimuli, classifying the stimuli as associated with a single user input event. - View Dependent Claims (49, 50)
-
-
66. A computer program product for filtering input events, the computer program product comprising:
-
a computer readable medium; and
computer program instructions, encoded on the medium, for controlling a processor to perform the operations of;
detecting, in a visual domain, a first plurality of input events resulting from user action;
detecting, in an auditory domain, a second plurality of input events resulting from user action;
for each detected event in the first plurality;
determining whether the detected event in the first plurality corresponds to a detected event in the second plurality; and
responsive to the detected event in the first plurality not corresponding to a detected event in the second plurality, filtering out the event in the first plurality. - View Dependent Claims (67, 68, 69, 70, 71, 72, 73, 74, 75)
-
-
76. A computer program product for classifying an input event, the computer program product comprising:
-
a computer readable medium; and
computer program instructions, encoded on the medium, for controlling a processor to perform the operations of;
receiving a visual stimulus, resulting from user action, in a visual domain;
receiving an acoustic stimulus, resulting from user action, in an auditory domain; and
generating a vector of visual features based on the received visual stimulus;
generating a vector of acoustic features based on the received acoustic stimulus;
comparing the generated vectors to user action descriptors for a plurality of user actions; and
responsive to the comparison indicating a match, outputting a signal indicating a recognized user action.
-
Specification