Passive touch system and method of detecting user input

• US 6,803,906 B1
• Filed: 07/05/2000
• Issued: 10/12/2004
• Est. Priority Date: 07/05/2000
• Status: Expired due to Term

• Current Assignee: Smart Technologies ULC (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Smart Technologies Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Morrison, Gerald, Singh, Manvinder
• Primary Examiner(s): Hjerpe, Richard
• Assistant Examiner(s): ZAMANI, ALI A
• Application Number:
  US09/610,481
• Time in Patent Office: 1,560 Days
• Field of Search: 345/173, 345/178, 345/156, 345/174, 345/157, 345/158, 345/175, 345/857, 345/862, 345/18.01, 345/181, 345/182, 345/183, 178/18.1, 178/18.01, 178/18.04
• US Class Current: 345/173
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421

Abstract

A passive touch system includes a passive touch surface and at least two cameras associated with the touch surface. The at least two cameras acquire images of the touch surface from different locations and have overlapping fields of view. A processor receives and processes images acquired by the at least two cameras to detect the existence of a pointer therein and to determine the location of the pointer relative to the touch surface. Actual pointer contact with the touch surface and pointer hover above the touch surface can be determined.

Claims

1. A passive touch system comprising:

a touch surface;

at least two cameras associated with said touch surface, said at least two cameras acquiring images of said touch surface from different locations and having overlapping fields of view;

a digital signal processor associated with each camera, the digital signal processors associated with said at least two cameras selecting pixel subsets of images acquired by said at least two cameras and processing pixel data acquired by the selected pixel subsets to generate pointer characteristic data when a pointer exists in said acquired images; and

a master digital signal processor in communication with said digital signal processors, said master digital signal processor receiving pointer characteristic data from said digital signal processors and triangulating the pointer characteristic data to determine the location of said pointer relative to said touch surface.

2. A passive touch system according to claim 1 wherein said digital signal processors generate pixel characteristic data when the pointer is in contact with said touch surface and when said pointer is hovering above said touch surface.

3. A passive touch system according to claim 2 wherein said selected pixel subsets are determined during an alignment routine.

4. A passive touch system according to claim 1 wherein said pointer characteristic data includes a median line of the pointer.

5. A passive touch system according to claim 4 wherein said pointer characteristic data is packaged by said digital signal processors into pointer information packets (PIPs).

6. A passive touch according to claim 1 wherein said touch surface is substantially rectangular and wherein a camera is located adjacent each corner of said touch surface.

7. A passive touch system according to claim 6, wherein each said camera comprises a CMOS digital camera having a selectable pixel array.

8. A passive touch system according to claim 7, wherein said selected pixel subsets include contiguous rows of pixels, and wherein the pixel data acquired by said contiguous rows of pixels is processed by said digital cameras at full resolution.

9. A passive touch system according to claim 7, wherein the selected pixel subset for each digital camera is aligned so that the selected pixel subset looks substantially along the plane of said touch surface.

10. A passive touch system according to claim 9, wherein the row of pixels in the selected pixel subset for each digital camera, that acquires pixel data corresponding to a tip of said pointer on said touch surface, is determined.

11. A passive touch system according to claim 10, wherein during said processing, said master digital signal processor triangulates pointer characteristic data associated with multiple pairs of digital cameras.

12. A passive touch system according to claim 11, wherein said master digital signal processor averages the triangulated pointer characteristic data to determine the location of said pointer relative to said touch surface.

13. A passive touch system according to claim 6, wherein during said processing, said master digital signal processor triangulates pointer characteristic data associated with multiple pairs of digital cameras.

14. A passive touch system according to claim 13, wherein said master digital signal processor averages the triangulated pointer characteristic data to determine the location of said pointer relative to said touch surface.

15. A passive touch system according to claim 1, wherein communication between said master digital signal processor and said digital signal processors is bidirectional.

16. A passive touch system according to claim 15, wherein said digital signal processors also communicate diagnostic data to said master digital signal processor.

17. A passive touch system as defined in claim 6 further including a computer coupled to said master digital signal processor, said computer receiving pointer location data from said master digital signal processor and processing the same.

18. A passive touch system according to claim 1, wherein each said at least two cameras comprises a CMOS digital camera having a selectable pixel array.

19. A passive touch system according to claim 18, wherein said selected pixel subsets include contiguous rows of pixels, and wherein the pixel data acquired by said contiguous rows of pixels is processed by said digital cameras at full resolution.

20. A passive touch system according to claim 18, wherein the selected pixel subset for each digital camera is aligned so that the selected pixel subset looks substantially along the plane of said touch surface.

21. A passive touch system according to claim 20, wherein the row of pixels in the selected pixel subset for each digital camera, that acquires pixel data corresponding to a tip of said pointer on said touch surface, is determined.

22. A touch system comprising:

at least two CMOS digital cameras associated with a touch surface, said at least two digital cameras acquiring images of said touch surface from different locations and having overlapping fields of view; and

a processor receiving and processing image data acquired by said at least two digital cameras to detect the existence of a pointer in said images and to determine the location of said pointer relative to said touch surface.

23. A touch system according to claim 22, wherein each of said at least two digital cameras has a selectable pixel array.

24. A touch system according to claim 23, wherein a subset of pixels in the selectable pixel array of each digital camera provides pixel data to said processor.

25. A touch system according to claim 24, wherein the pixel subset of each digital camera includes contiguous rows of pixels, and wherein pixel data acquired by said contiguous rows of pixels is processed at full resolution.

26. A touch system according to claim 24, wherein the pixel subset of each digital camera is aligned so that the pixel subset of each digital camera looks substantially along the plane of said touch surface.

27. A touch system according to claim 26, wherein the row of pixels in the pixel subset of each digital camera that acquires pixel data corresponding to a tip of said pointer on said touch surface is determined to enable said processor to determine pointer contact with said touch surface and pointer hover over said touch surface.

28. A touch system according to claim 22, wherein said touch surface is substantially rectangular wherein a digital camera is positioned adjacent each corner of said touch surface.

29. A touch system according to claim 28, wherein each said digital camera has a selectable pixel array.

30. A touch system according to claim 29, wherein a subset of pixels in the selectable pixel array of each digital camera provides pixel data to said processor.

31. A touch system according to claim 29, wherein the pixel subset of each digital camera includes contiguous rows of pixels, and wherein pixel data acquired by said contiguous rows of pixels is processed at full resolution.

32. A touch system according to claim 31, wherein the pixel subset of each digital camera is aligned so that the pixel subset of each digital camera looks substantially along the plane of said touch surface.

33. A touch system according to claim 32, wherein the row of pixels in the pixel subset of each digital camera, that acquires pixel data corresponding to a tip of said pointer on said touch surface is determined to enable said processor to detect pointer contact with said touch surface and pointer hover over said touch surface.

34. A touch system according to claim 28, wherein said processor triangulates image data from multiple pairs of digital cameras to determine the location of the pointer relative to said touch surface.

35. A touch system according to claim 34, wherein said processor includes a plurality of processing stages, said processing stages processing image data in a manner to stage bandwidth.

36. A touch system according to claim 35, wherein one of said processing stages includes a plurality of digital signal processors each associated with a respective one of said digital cameras, said digital signal processors processing pixel data from pixel subsets of said digital cameras and generating pointer parameter data.

37. A touch system according to claim 36, wherein a second of said processing stages includes a master digital signal processor receiving said pointer parameter data from said digital signal processors, said master digital signal processor triangulating said pointer parameter data.

38. A touch system according to claim 37, wherein a third of said processing stages includes a personal computer receiving the location of said pointer relative to said touch surface from said master digital signal processor.

39. A touch system according to claim 38, wherein communication between said master digital signal processor and said digital signal processors is bi-directional.

40. A touch system according to claim 39, wherein said digital signal processors also communicate digital camera diagnostic data to said master digital signal processor.

41. A touch system according to claim 28, wherein said digital cameras are arranged relative to said touch surface so that the fields of view thereof extend beyond at least one peripheral edge of said touch surface.

42. A touch system according to claim 41, wherein said processor includes a plurality of processing stages, said processing stages processing image data in a manner to stage bandwidth.

43. A touch system according to claim 42 wherein one of said processing stages includes a plurality of digital signal processors each associated with a respective one of said digital cameras, said digital signal processors processing pixel data from pixel subsets of said digital cameras and generating pointer parameter data.

44. A touch system according to claim 43, wherein a second of said processing stages includes a master digital signal processor receiving said pointer parameter data from said digital signal processors, said master digital signal processor triangulating said pointer parameter data.

45. A touch system according to claim 44, wherein a third of said processing stages includes a personal computer receiving the location of said pointer relative to said touch surface from said master digital signal processor.

46. A touch system comprising:

at least two optical recording devices associated with a touch surface, said at least two optical recording devices acquiring images of said touch surface from different locations and having overlapping fields of view; and

a processor receiving and processing image data acquired by said at least two optical recording devices to detect the existence of a pointer in said images and to determine the location of said pointer relative to said touch surface, wherein said processor includes first and second processing stages, said first processing stage processing pixel data from said at least two optical recording devices, said second processing stage processing image data from said first processing stage to determine the location of the pointer.

47. A touch system according to claim 46, wherein said first processing stage includes a plurality of digital signal processors each associated with a respective one of said optical recording devices, said digital signal processors processing pixel data from pixel subsets of said optical recording devices and generating pointer parameter data.

48. A touch system according to claim 47 wherein, said second processing stage includes a master digital signal processor receiving said pointer parameter data from said digital signal processors, said master digital signal processor triangulating said pointer parameter data.

49. A touch system according to claim 48, wherein said processor further includes a third processing stage comprising a personal computer receiving the location of said pointer relative to said touch surface from said master digital signal processor.

50. A touch system according to claim 49, wherein communication between said master digital signal processor and said digital signal processors is bi-directional.

51. A touch system according to claim 50, wherein said digital signal processors also communicate optical recording device diagnostic data to said master digital signal processor.

52. A touch system according to claim 48, wherein said master digital signal processor further calculates the velocity of said pointer as said pointer is brought towards said touch surface.

53. A touch system according to claim 52, wherein said velocity is calculated by determining the product of the change in pointer tip position within said pixel subset in successive images acquired by said digital cameras and the frame rates of the digital cameras.

54. A touch system according to claim 48, wherein said master digital signal processor further calculates the angle of said pointer as said pointer is brought towards said touch surface.

55. A touch system according to claim 54, wherein said angle is determined using the position of said pointer in extreme rows of pixels in said pixel subsets and the number of pixel rows in said pixel subsets.

56. A touch system according to claim 46, wherein said touch surface is substantially rectangular and wherein optical recording devices are positioned adjacent each corner of said touch surface, each optical recording device being arranger relative to said touch surface so that the field of view thereof extends beyond at least one peripheral edge of said touch surface.

57. A touch system according to claim 56, wherein each said optical recording device comprises a CMOS digital camera having a selectable pixel array.

58. A touch system according to claim 57, wherein said first processing stage includes a plurality of digital signal processors each associated with a respective one of said digital cameras, said digital signal processors processing pixel data from pixel subsets of said digital cameras and generating pointer parameter data.

59. A touch system according to claim 58, wherein said second processing stage includes a master digital signal processor receiving said pointer parameter data from said digital signal processors, said master digital signal processor triangulating said pointer parameter data.

60. A touch system according to claim 59, wherein said processor further includes a third processing stage comprising a personal computer receiving the location of said pointer relative to said touch surface from said master digital signal processor.

61. A scalable touch system comprising:

a projector to project an image onto a touch surface;

at least two digital cameras associated with said touch surface, said at least two digital cameras acquiring images of said touch surface from different locations and having overlapping fields of view; and

a processor receiving and processing image data acquired by said at least two digital cameras to detect the existence of a pointer in said images and to determine the location of said pointer relative to said touch surface, the location of said pointer being used to update said projected image, wherein each of said digital cameras includes a pixel array proportional to the number of pixels in said projected image, and wherein said processor is able to resolve the location of said pointer relative to said touch surface with sufficient accuracy with respect to the pixels in said projected image.

62. A touch system according to claim 61, wherein each said at least two digital cameras comprises a CMOS digital camera having a selectable pixel array.

63. A touch system according to claim 62, further including a rectangular bezel surrounding said touch surface, a CMOS digital camera being positioned at each corner of said bezel.

64. A touch system according to claim 63, wherein said processor triangulates image data from multiple pairs of digital cameras to determine the location of the pointer relative to said touch surface.

65. A touch system according to claim 64, wherein said digital cameras are arranged relative to said touch surface so that the fields of view thereof extend beyond at least one peripheral edge of said bezel.

66. A touch system according to claim 65, wherein said processor includes a plurality of processing stages, said processing stages processing image data in a manner to stage bandwidth.

67. A touch system according to claim 66, wherein one of said processing stages includes a plurality of digital signal processors each associated with a respective one of said digital cameras, said digital signal processors processing pixel data from pixel subsets of said digital cameras and generating pointer parameter data.

68. A touch system according to claim 67, wherein a second of said processing stages includes a master digital signal processor receiving said pointer parameter data from said digital signal processors, said master digital signal processor triangulating said pointer parameter data.

69. A touch system according to claim 68, wherein a third of said processing stages includes a personal computer receiving the location of said pointer relative to said touch surface from said master digital signal processor.

70. A touch system according to claim 64, wherein a subset of pixels in the selectable pixel array of each digital camera provides pixel data to said processor.

71. A touch system according to claim 70, wherein each pixel subset includes contiguous rows of pixels and wherein the pixel data acquired by said contiguous rows of pixels is processed at full resolution.

72. A touch system according to claim 70, wherein the pixel subset of each digital camera is aligned so that the pixel subset of each digital camera looks substantially along the plane of said touch surface.

73. A touch system according to claim 72, wherein the row of pixels in the pixel subset of each digital camera, that acquires pixel data corresponding to a tip of said pointer tip on said touch surface, is determined to enable said processor to detect pointer contact with said touch surface and pointer hover over said touch surface.

74. A touch system comprising:

at least two CMOS digital cameras associated with a touch surface and having overlapping fields of view, each of said at least two CMOS digital cameras having an array of pixels, said at least two CMOS digital cameras acquiring images of said touch surface from different locations substantially simultaneously; and

a processor receiving and processing pixel data acquired by a window of contiguous pixels within the pixel array of said at least two CMOS digital cameras to detect the existence of a pointer in said images and to triangulate the location of said pointer relative to said touch surface.

75. A touch system according to claim 74, wherein the window of contiguous pixels used to provide said pixel data from each digital camera is adjustable along said pixel array.

76. A touch system according to claim 75, wherein said pixel data is processed at full resolution.

77. A touch system according to claim 75, wherein the window of pixels of each digital camera is aligned so that pixel data corresponding to a tip of said pointer on said touch surface is acquired.

78. A touch system according to claim 77, wherein the row of pixels in the window of each digital camera, that acquires the pixel data corresponding to said pointer tip on said touch surface, is determined to enable said processor to determine pointer contact with said touch surface and pointer hover over said touch surface.

79. A touch system according to claim 78, further comprising a bezel surrounding said touch surface, said bezel supporting a digital camera adjacent each corner thereof, said digital cameras being arranged so that the field of view of each digital camera extends beyond at least one peripheral edge of said bezel.

80. A touch system according to claim 79, wherein said processor triangulates pixel data from multiple pairs of digital cameras to determine the location of the pointer relative to said touch surface.

81. A touch system according to claim 80, wherein said processor includes a plurality of processing stages, said processing stages processing image data in a manner to stage bandwidth.

82. A touch system according to claim 81, wherein one of said processing stages includes a plurality of digital signal processors each associated with a respective one of said digital cameras, said digital signal processors processing pixel data from pixel subsets of said digital cameras and generating pointer parameter data.

83. A touch system according to claim 82, wherein a second of said processing stages includes a master digital signal processor receiving said pointer data from said digital signal processors, said master digital signal processor triangulating said pointer data.

84. A touch system according to claim 83, wherein a third of said processing stages includes a personal computer receiving the location of said pointer relative to said touch surface from said master digital signal processor.

85. A method of aligning a plurality of digital cameras in a touch system, each digital camera having a selectable pixel array with respect to a touch system touch surface on which a pointer contact is made, said method comprising the steps of:

acquiring an image of said touch surface;

examining pixels within a subset of said pixel array to determine if the pixel subset is substantially looking along the plane of said touch surface; and

if not

adjusting the position of said pixel subset and repeating said acquiring and said examining steps until a pixel subset is determined that looks substantially along the plane of said touch surface.

86. The method of claim 85, wherein during said examining, pixels within said pixel subset are examined to determine if said pixels include data referencing said touch surface.

87. The method of claim 86, wherein said data comprises a tip of said pointer in contact with said touch surface.

88. The method of claim 86, further comprising the step of determining the row of pixels within said pixel subset when data corresponding to the tip of said pointer in contact with said touch surface exists.

89. An imaging assembly comprising:

a substantially rectangular bezel to surround a display surface; and

a digital camera mounted adjacent each corner of said bezel, said digital cameras being oriented to capture overlapping images of said display surface.

90. An imaging assembly according to claim 89, wherein said digital cameras comprise CMOS digital cameras.

91. An imaging assembly according to claim 90, wherein each said digital camera has a selectable pixel array.

92. An imaging assembly according to claim 91, wherein said digital cameras are oriented so that the field of view of each digital camera extends beyond a peripheral edge of said bezel.

93. An imaging assembly according to claim 92, further including a processor associated with each of said digital cameras, each digital camera and associated processor being mounted on a common board.

94. An imaging assembly according to claim 93, wherein a subset of pixels in the selectable pixel array of each digital camera provides pixel data to said associated processor.

95. An imaging assembly according to claim 94, wherein each pixel subset includes contiguous rows of pixels.

96. An imaging assembly according to claim 95, wherein the pixel subset of each digital camera is aligned so that the pixel subset looks substantially along the plane of said touch surface.

97. An imaging assembly according to claim 96, wherein the row of pixels in the pixel subset of each digital camera, that acquires pixel data corresponding to a tip of said pointer on said touch surface, is determined to enable said processor to detect pointer contact with said touch surface and pointer hover over said touch surface.

98. A touch system comprising:

a substantially rectangular touch surface; and

a digital camera mounted adjacent each corner of said touch surface, said digital cameras being oriented to capture overlapping images of said touch surface.

99. A touch system according to claim 98, wherein said digital cameras comprise CMOS digital cameras.

100. A touch system according to claim 99, wherein each said digital camera has a selectable pixel array.

101. A touch system according to claim 100, wherein said digital cameras are oriented so that the field of view of each digital camera extends beyond a peripheral edge of said touch surface.

102. A touch system according to claim 101, further including a processor associated with each of said digital cameras, each digital camera and associated processor being mounted on a common board.

103. A touch system according to claim 102, wherein a subset of pixels in the selectable pixel array of each digital camera provides pixel data to said associated processor.

104. A touch system according to claim 103, wherein each pixel subset includes contiguous rows of pixels.

105. A touch system according to claim 104, wherein the pixel subset of each digital camera is aligned so that the pixel subset of each digital camera looks substantially along the plane of said touch surface.

106. A touch system according to claim 105, wherein the row of pixels in the pixel subset of each digital camera, that acquires pixel data corresponding to a tip of said pointer on said touch surface, is determined to enable said processor to detect pointer contact with said touch surface and pointer hover over said touch surface.

107. A method of detecting the position of a pointer relative to a touch surface comprising the steps of:

acquiring multiple images of a pointer relative to said touch surface;

selecting pixel subsets of said acquired images; and

processing pixel data acquired by said pixel subsets to detect the existence of said pointer therein and to determine the location of said pointer relative to said touch surface using triangulation.

108. The method of claim 107 wherein during said processing step, the pixel data is processed to determine when said pointer is in contact with said touch surface and when said pointer is hovering over said touch surface.

109. The method of claim 108 wherein during said processing step the existence of said pointer is determined by calculating median lines of the pointer and wherein the location of said pointer is determined by calculating the intersection point of median lines and using triangulation to determine the coordinates of said intersection point.