Recognizing gestures and using gestures for interacting with software applications
First Claim
1. A method for using a gesture in connection with an interactive display system on which images are displayed and objects disposed on or proximate thereto are optically detectable, comprising the steps of:
- (a) forming an image with light reflected from an object that is disposed above a display surface of the interactive display system, the light that is thus reflected being detected after passing through the display surface;
(b) processing the image to recognize a gesture that was made by the user using the object, said gesture being recognized in response to one of;
(i) identifying a shape corresponding to the object within the image that is detected; and
(ii) detecting at least one connected component within the image, said connected component comprising adjacent pixels having an intensity above a predefined threshold; and
(c) as a function of the gesture that was recognized, automatically producing a response that has previously been associated with the gesture, said response changing a state of the interactive display system.
3 Assignments
0 Petitions
Accused Products
Abstract
An interactive display table has a display surface for displaying images and upon or adjacent to which various objects, including a user'"'"'s hand(s) and finger(s) can be detected. A video camera within the interactive display table responds to infrared (IR) light reflected from the objects to detect any connected components. Connected component correspond to portions of the object(s) that are either in contact, or proximate the display surface. Using these connected components, the interactive display table senses and infers natural hand or finger positions, or movement of an object, to detect gestures. Specific gestures are used to execute applications, carryout functions in an application, create a virtual object, or do other interactions, each of which is associated with a different gesture. A gesture can be a static pose, or a more complex configuration, and/or movement made with one or both hands or other objects.
-
Citations
46 Claims
-
1. A method for using a gesture in connection with an interactive display system on which images are displayed and objects disposed on or proximate thereto are optically detectable, comprising the steps of:
-
(a) forming an image with light reflected from an object that is disposed above a display surface of the interactive display system, the light that is thus reflected being detected after passing through the display surface;
(b) processing the image to recognize a gesture that was made by the user using the object, said gesture being recognized in response to one of;
(i) identifying a shape corresponding to the object within the image that is detected; and
(ii) detecting at least one connected component within the image, said connected component comprising adjacent pixels having an intensity above a predefined threshold; and
(c) as a function of the gesture that was recognized, automatically producing a response that has previously been associated with the gesture, said response changing a state of the interactive display system. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23)
-
-
24. An interactive display system that responds to a gesture made with an object, comprising:
-
(a) a display surface having an interactive side on or adjacent to which an object can be placed and manipulated, and an opposite side;
(b) a light source that directs light toward the opposite side of the display surface and through the display surface, to the interactive side;
(c) a light sensor disposed to receive and sense light reflected back from an object through the display surface, producing a signal corresponding to an image of an object that is in contact or proximate to the display surface;
(d) a processor in communication with the light sensor; and
(e) a memory in communication with the processor, the memory storing data and machine instructions that cause the processor to carry out a plurality of functions, including;
(i) processing the image to recognize a gesture that was made by the user using the object, said gesture being recognized in response to one of;
(1) identifying a shape corresponding to the object within the image; and
(2) detecting at least one connected component within the image, said connected component comprising adjacent pixels having an intensity above a predefined threshold; and
(ii) as a function of the gesture that was recognized, automatically producing a response that has previously been associated with the gesture in the memory, said response changing a state of the interactive display system. - View Dependent Claims (25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46)
-
Specification