METHOD FOR MANIPULATING A GRAPHICAL USER INTERFACE AND INTERACTIVE INPUT SYSTEM EMPLOYING THE SAME

• US 20130055143A1
• Filed: 08/31/2012
• Published: 02/28/2013
• Est. Priority Date: 08/31/2011
• Status: Abandoned Application

• Current Assignee: Smart Technologies ULC (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Smart Technologies ULC (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Martin, David, Hill, Douglas, Tse, Edward, Segelken, Wendy
• Application Number:
  US13/601,429
• Publication Number:
  US 20130055143A1
• Time in Patent Office: Days
• US Class Current: 715/779
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

A method comprises capturing at least one image of a three-dimensional (3D) space disposed in front of a display surface and processing the captured at least one image to detect a pointing gesture made by a user within the three-dimensional (3D) space and the position on the display surface to which the pointing gesture is aimed.

Claims

1. A method comprising:

capturing at least one image of a three-dimensional (3D) space disposed in front of a display surface; and

processing the captured at least one image to detect a pointing gesture made by a user within the three-dimensional (3D) space and the position on the display surface to which the pointing gesture is aimed.

2. The method of claim 1 wherein said processing comprises:

identifying at least two reference points on the user associated with the pointing gesture;

calculating a 3D vector connecting the at least two reference points; and

extrapolating the 3D vector towards the display surface to identify the position thereon.

3. The method of claim 1 further comprising:

displaying an indicator on the display surface at the position.

4. The method of claim 3 wherein the indicator is one of a temporary indicator and a permanent indicator.

5. The method of claim 4 wherein the indicator is selected based on whether the position is aimed at an active area or inactive area of an image displayed on the display surface.

6. The method of claim 2 wherein the at least two reference points comprise the user'"'"'s hand and an eye or the user'"'"'s hand and elbow.

7. The method of claim 2 wherein the processing further comprises:

calculating a distance between the user and the display surface.

8. The method of claim 7 further comprising:

comparing the distance between the user and the display surface to a threshold.

9. The method of claim 8 wherein the distance between the user and the display surface determines the at least two reference points on the user that are identified.

10. The method of claim 9 wherein when the distance between the user and the display surface is greater than the threshold, one of the at least two reference points on the user corresponds to the user'"'"'s eyes.

11. The method of claim 10 wherein the other of the at least two identified reference points on the user corresponds to the user'"'"'s hand.

12. The method of claim 9 wherein when the distance between the user and the display surface is less than the threshold, one of the at least two identified reference points on the user corresponds to the user'"'"'s elbow.

13. The method of claim 12 wherein the other of the at least two identified reference points on the user corresponds to the user'"'"'s hand.

14. The method of claim 3 wherein the size of the indicator is dependent on the distance between the user and the display surface.

15. The method of claim 14 wherein when the distance between the user and the display surface is greater than a threshold, the indicator is displayed in a large format.

16. The method of claim 15 wherein when the distance between the user and the display surface is less than the threshold, the indicator is displayed in a small format.

17. The method of claim 7 further comprising:

comparing the distance between the user and the display surface to a threshold.

18. The method of claim 17 wherein when the distance between the user and the display surface is greater than the threshold, displaying an indicator on the display surface in a large format.

19. The method of claim 17 wherein when the distance between the user and the display surface is less than the threshold, displaying an indicator on the display surface in a small format.

20. An interactive input system comprising:

a display surface;

at least one imaging device configured to capture images of a three-dimensional (3D) space disposed in front of the display surface; and

processing structure configured to process the captured images to detect a user making a pointing gesture towards the display surface and the position on the display surface to which the pointing gesture is aimed.

21. The interactive input system of claim 20 wherein the processing structure is configured to identify at least two reference points on the user, calculate a 3D vector connecting the at least two reference points, and extrapolate the 3D vector towards the display surface to identify the position on the display surface.

22. The interactive input system of claim 21 wherein the processing structure is configured to display an indicator on the display surface at the position.

23. The interactive input system of claim 22 wherein the indicator is one of a temporary indicator and a permanent indicator.

24. The interactive input system of claim 23 wherein the indicator is selected based on whether the position is aimed at an active area or inactive area of an image displayed on the display surface.

25. The interactive input system of claim 22 wherein the processing structure is configured to calculate a distance between the user and the display surface.

26. The interactive input system of claim 25 wherein the distance between the user and the display surface determines the at least two reference points on the user that are identified.

27. The interactive input system of claim 26 wherein when the distance between the user and the display surface is greater than a threshold, one of the at least two identified reference points corresponds to a position of the user'"'"'s eyes.

28. The interactive input system of claim 27 wherein the indicator is displayed in a large format.

29. The interactive input system of claim 26 wherein when the distance between the user and the display surface is less than a threshold, one of the at least two identified reference points corresponds to a position of the user'"'"'s elbow.

30. The interactive input system of claim 29 wherein the indicator is displayed in a small format.

31. The interactive input system of claim 28 wherein the other of the at least two identified reference points corresponds to a position of the user'"'"'s hand.

32. The interactive input system of claim 30 wherein the other of the at least two identified reference points corresponds to a position of the user'"'"'s hand.

33. A method of manipulating a graphical user interface (GUI) displayed on a display surface comprising:

receiving an input event from an input device;

processing the input event to determine the location of the input event and the type of the input event;

comparing at least one of the location of the input event and the type of the input event to defined criteria; and

manipulating the GUI based on the result of the comparing.

34. The method of claim 33 further comprising:

partitioning the GUI into an active control area and an inactive area.

35. The method of claim 34 wherein said comparing is performed to determine if the location of the input event on the GUI corresponds to the active control area or the inactive area.

36. The method of claim 35 wherein the manipulating comprises applying an indicator to the GUI for display on the display surface at the location of the input event when the location of the input event corresponds to the inactive area.

37. The method of claim 36 wherein the comparing is performed to determine if the type of input event is a touch input event.

38. The method of claim 37 wherein when the input event is a touch input event, the comparing is further performed to determine a pointer type associated with the input event.

39. The method of claim 38 wherein the comparing is performed to determine if the pointer type associated with the input event is a first pointer type or a second pointer type.

40. The method of claim 39 wherein the manipulating comprises applying a temporary indicator to the GUI for display on the display surface at the location of the input event if the pointer type is the first pointer type.

41. The method of claim 40 wherein the first pointer type is a user'"'"'s finger.

42. The method of claim 39 wherein the manipulating comprises applying a permanent indicator to the GUI for display on the display surface at the location of the input event if the pointer type is the second pointer type.

43. The method of claim 42 wherein the second pointer type is a pen tool.

44. The method of claim 35 wherein when the location of the input event on the GUI corresponds to the active control area, the comparing is performed to determine a type of active graphic object associated with the location of the input event.

45. The method of claim 44 wherein the manipulating comprises performing an update on the GUI displayed on the display surface based on the type of active graphic object.

46. The method of claim 45 wherein the type of graphic object is one of a menu, a toolbar, and a button.

47. The method of claim 37 wherein when the input event is a touch input event, the comparing is further performed to determine a pointer size associated with the input event.

48. The method of claim 47 wherein the further comparing is performed to compare the pointer size to a threshold.

49. The method of claim 48 wherein when the pointer size is less than the threshold, the indicator is of a first type.

50. The method of claim 48 wherein when the pointer size is less than the threshold, the indicator is of a small size.

51. The method of claim 48 wherein when the pointer is greater than the threshold, the indicator is of a second type.

52. The method of claim 48 wherein when the pointer is greater than the threshold, the indicator is of a large size.

53. The method of claim 38 wherein when the pointer type is a physical object and the location of the input event obstructs at least a portion of the active control area, the manipulating comprises moving the obstructed portion of the active control area to a new location on the GUI such that no portion of the active control area is obstructed by the physical object.

54. An interactive input system comprising:

a display surface on which a graphical user interface (GUI) is displayed;

at least one input device; and

processing structure configured to receive an input event from the at least one input device, determine the location of the input event and the type of the input event, compare at least one of the location of the input event and the type of the input event to defined criteria, and manipulate the GUI based on the result of the comparing.

55. The interactive input system of claim 54 wherein the processing structure partitions the GUI into an active control area and an inactive area.

56. The interactive input system of claim 55 wherein the processing structure compares the location of the input event to the defined criteria to determine if the location of the input event on the GUI corresponds to the active control area or the inactive area.

57. The interactive input system of claim 56 wherein during the manipulating, the processing structure applies an indicator to the GUI for display on the display surface at the location of the input event when the location of the input event corresponds to the inactive area.

58. The interactive input system of claim 56 wherein the processing structure compares the input event to the defined criteria to determine if the type of input event is a touch input event.

59. The interactive input system of claim 58 wherein when the input event is a touch input event, the processing structure determines a pointer type associated with the input event.

60. The interactive input system of claim 59 wherein the processing structure compares the input event to the defined criteria to determine if the pointer type associated with the input event is a first pointer type or a second pointer type.

61. The interactive input system of claim 60 wherein during the manipulating, the processing structure applies a temporary indicator to the GUI for display on the display surface at the location of the input event when the pointer type is the first pointer type.

62. The interactive input system of claim 61 wherein the first pointer type is a user'"'"'s finger.

63. The interactive input system of claim 60 wherein during the manipulating, the processing structure applies a permanent indicator to the GUI for display on the display surface at the location of the input event when the pointer type is the second pointer type.

64. The interactive input system of claim 63 wherein the second pointer type is a pen tool.

65. The interactive input system of claim 56 wherein when the location of the input event on the GUI corresponds to the active control area, the processing structure compares the input event to the defined criteria to determine a type of active graphic object associated with the location of the input event.

66. The interactive input system of claim 65 wherein during the manipulating, the processing structure performs an update on the GUI displayed on the display surface based on the type of active graphic object.

67. The interactive input system of claim 65 wherein the type of graphic object is one of a menu, a toolbar, and a button.

68. The interactive input system of claim 58 wherein when the input event is a touch input event, the processing structure determines a pointer size associated with the input event.

69. The interactive input system of claim 68 wherein the processing structure compares the pointer size to a threshold.

70. The interactive input system of claim 69 wherein when the event that the pointer size is less than the threshold, the indicator is of a first type or a small size.

71. The interactive input system of claim 69 wherein when the pointer is greater than the threshold, the indicator is of a second type or a large size.

72. The interactive input system of claim 58 wherein when the pointer type is a physical object and the location of the input event obstructs at least a portion of the active control area, during the manipulating the processing structure moves the obstructed portion of the active control area to a new location on the GUI such that no portion of the active control area is obstructed by the physical object.

73. A method of manipulating a shared graphical user interface (GUI) displayed on a display surface of at least two client devices, one of the client devices being a host client device, the at least two client devices participating in a collaboration session, the method comprising:

receiving, at the host client device, an input event from an input device associated with an annotator device of the collaboration session;

processing the input event to determine the location of the input event and the type of the input event;

comparing at least one of the location of the input event and the type of the input event to defined criteria; and

manipulating the shared GUI based on the results of the comparing.

74. The method of claim 73 wherein the manipulating comprises applying an indicator to the shared GUI for display on the display surface of each of the at least two client devices.

75. The method of claim 73 further comprising determining if the host client device is the annotator device.

76. The method of claim 75 wherein when the host client device is the annotator device, applying an indicator to the shared GUI for display on the display surface of each of the at least two client devices with the exception of the host client device.

77. A method of applying an indicator to a graphical user interface (GUI) displayed on a display surface, the method comprising:

receiving an input event from an input device;

determining characteristics of said input event, the characteristics comprising at least one of the location of the input event and the type of the input event;

determining if the characteristics of the input event satisfies defined criteria; and

manipulating the GUI if the defined criteria is satisfied.

78. The method of claim 77 wherein the manipulating comprises applying an indicator to the GUI for display on the display surface.

79. The method of claim 77 wherein determining if the characteristics of the input event satisfies the defined criteria comprises comparing at least one of the characteristics of the input event to the defined criteria.

80. A method of processing an input event comprising:

receiving an input event from an input device;

determining characteristics of the input event, the characteristics comprising at least one of the location of the input event and the type of the input event;

determining an application program to which the input event is to be applied;

determining whether the characteristics of the input event satisfies defined criteria; and

sending the input event to the application program if the defined criteria is satisfied.