Recognizing gestures and using gestures for interacting with software applications

• US 7,519,223 B2
• Filed: 06/28/2004
• Issued: 04/14/2009
• Est. Priority Date: 06/28/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wobbrock, Jacob O., Dehlin, Joel P., Wilson, Andrew D., Chen, Christina Summer, Horvitz, Eric J., Robbins, Daniel C., Hinckley, Kenneth P.
• Primary Examiner(s): Strege; John B
• Application Number:
  US10/880,167
• Publication Number:
  US 20060010400A1
• Time in Patent Office: 1,751 Days
• Field of Search: 382/103, 382/181, 382/203, 345/173, 345/175, 348/77, 715/863
• US Class Current: 382/203
• CPC Class Codes: G06F 2203/0382 Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0354 with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

An interactive display table has a display surface for displaying images and upon or adjacent to which various objects, including a user'"'"'s hand(s) and finger(s) can be detected. A video camera within the interactive display table responds to infrared (IR) light reflected from the objects to detect any connected components. Connected component correspond to portions of the object(s) that are either in contact, or proximate the display surface. Using these connected components, the interactive display table senses and infers natural hand or finger positions, or movement of an object, to detect gestures. Specific gestures are used to execute applications, carryout functions in an application, create a virtual object, or do other interactions, each of which is associated with a different gesture. A gesture can be a static pose, or a more complex configuration, and/or movement made with one or both hands or other objects.

Claims

1. A method for using a gesture in connection with an interactive display system on which images are displayed and objects disposed on or proximate thereto are optically detectable, comprising the steps of:

(a) forming an image with light reflected from an object that is disposed above a display surface of the interactive display system, the light that is thus reflected being detected after passing through the display surface;

(b) processing the image to distinguish between touch objects physically touching the display surface and hover objects proximate to the display surface and to recognize a gesture that was made by the user using the object, said gesture being recognized in response to detecting at least one connected component within the image, said connected component comprising adjacent pixels having an intensity within a predefined threshold; and

(c) as a function of the gesture that was recognized, automatically producing a response that has previously been associated with the gesture, said response changing a state of the interactive display system.

2. The method of claim 1, wherein the object comprises at least a portion of an appendage of a user'"'"'s body.

3. The method of claim 1, wherein the step of identifying the shape comprises the step of comparing the shape to a predefined template.

4. The method of claim 1, wherein the object is manipulated by a user grasping the object in the user'"'"'s hand.

5. The method of claim 1, wherein the gesture includes a movement of the object along a generally defined path.

6. The method of claim 2, wherein the gesture comprises a pose of the at least the portion of the appendage adjacent to the display surface, wherein the at least a portion the appendage results in a touch object and a hover object.

7. The method of claim 1, wherein the at least one connected component includes at least one touch connected component corresponding to a contact of the display surface by the object and at least one hover connected component.

8. The method of claim 2, wherein the at least one connected component corresponds to at least one finger as a touch connected component and a hand of the user acting as a hover connected component.

9. The method of claim 1, wherein the response that is automatically produced comprises the step of executing a software application associated with the gesture, on the interactive display system.

10. The method of claim 9, wherein the gesture defines an alpha character that relates to the software application.

11. The method of claim 1, wherein the response that is automatically produced comprises the step of executing a function within an application that is being executed by the interactive display system.

12. The method of claim 11, wherein the response further comprises the step of enabling a user to move the object to indicate an extent by which a variable relating to the function is changed when the function is executed.

13. The method of claim 1, wherein the response that is automatically produced comprises the step of selecting a region defined by the gesture on the display surface of the interactive display system.

14. The method of claim 1, wherein the response that is automatically produced comprises the step of selecting one of an image and a virtual object that is presented on the display surface.

15. The method of claim 14, wherein the response further comprises the step of enabling a user to resize the one of the image and the virtual object by moving the object over the display surface.

16. The method of claim 1, wherein the response comprises the step of presenting a virtual object on the display surface, the virtual object being associated with the gesture.

17. The method of claim 2, wherein the step of detecting at least one connected component comprises the step of detecting a plurality of touch connected components and at least one hover connected component to generally determine an orientation of the appendage of the user, so as to enable the gesture to be correctly recognized.

18. The method of claim 17, wherein the plurality of touch connected components corresponds to fingers of the user that are in contact with the display surface, and wherein the at least one hover connected component corresponds to a hand of the user that is proximate to the display surface.

19. The method of claim 17, wherein the plurality of touch connected components are used to identify multiple fingers that are in contact with the display surface, and to determine that the multiple fingers belong to one hand of a user.

20. The method of claim 17, further comprising the steps of:

(a) analyzing temporal patterns of the plurality of touch connected components as the touch connected components change over time;

(b) identifying groupings of multiple fingers that are in contact with the display surface; and

(c) determining that the multiple fingers are coupled to a hand of a specific user, based upon the orientation of the connected components and their temporal patterns.

21. The method of claim 17, further comprising the steps of:

(a) analyzing temporal patterns of a plurality of sets of touch connected components as the touch connected components change over time;

(b) identifying appendages of each of a plurality of different users, where the appendages are hovering adjacent to the display surface; and

(c) determining that groupings of multiple fingers in contact with the display surface belong to specific ones of the appendages, and thus, belong to specific users, based on an orientation of each appendage, and on the temporal patterns.

22. The method of claim 21, wherein the step of determining comprises the step of determining that an appendage and a grouping of multiple fingers belongs to a specific user based upon knowledge of where the specific user is located around the display surface.

23. A computer readable storage medium storing machine instructions for performing a method for using a gesture in connection with an interactive display system on which images are displayed and objects disposed on or proximate thereto are optically detectable, the method comprising:

(a) processing an image to distinguish between touch objects physically touching a display surface and hover objects proximate to the display surface and to recognize a gesture that was made by the a user using the object, said gesture being recognized in response to detecting a plurality of touch connected components and at least one hover connected component to generally determine an orientation of an appendage of the user, said connected components comprising adjacent pixels having an intensity above a predefined threshold; and

(b) as a function of the gesture that was recognized, automatically producing a response that has previously been associated with the gesture, said response changing a state of the interactive display system;

(c) analyzing temporal patterns of a plurality of sets of touch connected components as the touch connected components change over time;

(d) identifying appendages of each of a plurality of different users, where the appendages are hovering adjacent to the display surface; and

(e) determining that groupings of multiple fingers in contact with the display surface belong to specific ones of the appendages, and thus, belong to specific users, based on an orientation of each appendage, and on the temporal patterns.

24. An interactive display system that responds to a gesture made with an object, comprising:

(a) a display surface having an interactive side on or adjacent to which an object can be placed and manipulated, and an opposite side;

(b) a light source that directs light toward the opposite side of the display surface and through the display surface, to the interactive side;

(c) a light sensor disposed to receive and sense light reflected back from an object through the display surface, producing a signal corresponding to an image of an object that is in contact or proximate to the display surface;

(d) a processor in communication with the light sensor; and

(e) a memory in communication with the processor, the memory storing data and machine instructions that cause the processor to carry out a plurality of functions, including;

(i) processing the image to distinguish between touch objects physically touching the display surface and hover objects proximate to the display surface and to recognize a gesture that was made by the user using the object, said gesture being recognized in response to detecting at least one connected component within the image, said connected component comprising adjacent pixels having an intensity above a predefined threshold; and

(ii) as a function of the gesture that was recognized, automatically producing a response that has previously been associated with the gesture in the memory, said response changing a state of the interactive display system.

25. The interactive display system of claim 24, wherein the object comprises at least a portion of an appendage of a user.

26. The interactive display system of claim 24, wherein the shape is identified by comparing the shape to a predefined template.

27. The interactive display system of claim 24, wherein the object is grasped in a hand of a user.

28. The interactive display system of claim 24, wherein the gesture includes a movement of the object along a generally defined path.

29. The interactive display system of claim 25, wherein the gesture comprises a pose of at least the portion of an appendage adjacent to the display surfaced, wherein the at least a portion the appendage results in a touch object and a hover object.

30. The interactive display system of claim 24, wherein the at least one connected component includes at least one touch connected component corresponding to a contact of the display surface by the object and at least one hover connected component.

31. The interactive display system of claim 25, wherein the at least one connected component corresponds to at least one finger as a touch connected component and a hand of a user acting as a hover connected component.

32. The interactive display system of claim 24, wherein the response that is automatically produced is executing a software application associated with the gesture, using the processor.

33. The interactive display system of claim 32, wherein the gesture defines an alpha character that relates to the software application.

34. The interactive display system of claim 24, wherein the response that is automatically produced comprises executing a function within a software application that is being executed by the processor.

35. The interactive display system of claim 34, wherein the response further comprises enabling a user to move the object to indicate an extent by which a variable relating to the function is changed when the function is executed by the processor.

36. The interactive display system of claim 24, wherein the response that is automatically produced comprises selecting a region defined by the gesture on the display surface.

37. The interactive display system of claim 24, wherein the response that is automatically produced comprises selecting one of an image and a virtual object that is presented display surface, as determined by the gesture.

38. The interactive display system of claim 37, wherein the response further comprises enabling a user to resize the one of the image and the virtual object by moving the object over the display surface.

39. The interactive display system of claim 24, wherein the response comprises presenting a virtual object on the display surface, the virtual object being associated with the gesture.

40. The interactive display system of claim 25, wherein the machine instructions cause the processor to detect a plurality of touch connected components and at least one hover connected component to generally determine an orientation of an appendage of a user, so as to enable the gesture to be correctly recognized.

41. The interactive display system of claim 40, wherein the plurality of touch connected components corresponds to fingers of a user that are in contact with the display surface, and wherein the at least one hover connected component corresponds to a hand of a user that is proximate to the display surface.

42. The interactive display system of claim 40, wherein the plurality of touch connected components are used to identify multiple fingers that are in contact with the display surface, and to determine that the multiple fingers belong to one hand of a user.

43. The interactive display system of claim 40, wherein the machine instructions further cause the processor to analyze temporal patterns of the plurality of touch connected components as the touch connected components change over time, to identify groupings of multiple fingers that are in contact with the display surface, and to determine that the multiple fingers are coupled to a hand of a specific user, based upon the orientation of the connected components and their temporal patterns.

44. The interactive display system of claim 40, wherein the machine instructions further cause the processor to analyze temporal patterns of a plurality of sets of touch connected components as the touch connected components change over time, to identify appendages of each of a plurality of different users, where the appendages are hovering adjacent to the display surface, and to determine that groupings of multiple fingers in contact with the display surface belong to specific ones of the appendages, and thus, belong to specific users, based on an orientation of each appendage, and on the temporal patterns.

45. The interactive display system of claim 44, wherein the machine instructions further cause the processor to determine that an appendage and a grouping of multiple fingers belongs to a specific user based upon knowledge of where the specific user is located around the display surface.

46. The interactive display system of claim 24, wherein the light source produces infrared light and the light sensor senses infrared light that is reflected from object.