Camera-based touch system

• US 20050077452A1
• Filed: 07/05/2001
• Published: 04/14/2005
• Est. Priority Date: 07/05/2000
• Status: Active Grant

• Current Assignee: Smart Technologies Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Smart Technologies Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Morrison, Gerald, Holmgren, Daivd
• Granted Patent: US 7,692,625 B2
• Time in Patent Office: Days
• US Class Current: 250/221
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421

Abstract

A camera-based touch system (50) includes a passive touch surface (60) and at least two cameras (63) associated surface. The at least two cameras (63) have overlapping fields of view (FOV) encompassing the touch surface. The at least two cameras (63) acquire images of the touch surface from different locations and generate image data. A processor (54) receives and processes image data generated by the at least two cameras to determine the location of the pointer relative to the touch surface when the pointer is captured in images acquired by the at least two cameras. Actual pointer contact with the touch surface sand pointer hover above the touch surface can be determined.

Claims

1. A camera-based touch system comprising:

at least two cameras associated with a passive touch surface and having overlapping fields of view encompassing said touch surface, said at least two cameras acquiring images of said touch surface from different locations and generating image data; and

a processor receiving and processing image data generated by said at least two cameras to determine the location of a pointer relative to said touch surface when said pointer is captured in images acquired by said at least two cameras.

2. A touch system according to claim 1 wherein said at least two cameras are digital cameras having fields of view looking generally along the plane of said touch surface.

3. A touch system according to claim 2 wherein the image data generated by each digital camera includes a pointer median line x and a pointer tip location z.

4. A touch system according to claim 3 wherein each of said digital cameras includes a pixel array having selectable pixel rows, pixel intensities of pixels in said selectable pixel rows being used during generation of said image data.

5. A touch system according to claim 4 wherein pixel intensities of pixels in a region of interest within said selectable pixel rows are used during generation of said image data.

6. A touch system according to claim 5 wherein each of said digital cameras packages said image data into packets for transmission to said processor to provide bandwidth economy.

7. A touch system according to claim 6 wherein each of said digital cameras includes a CMOS image sensor and a digital signal processor, said digital signal processor receiving image output from said image sensor and executing a find pointer routine to determine if a pointer is in each image acquired by said digital camera, and if so the median line x of said pointer.

8. A touch system according to claim 7 wherein during said find pointer routine, said digital signal processor builds a vertical intensity histogram including columns of pixel intensities representing differences between an acquired image and a background image, the column of said vertical intensity-histogram having the largest pixel intensity above a threshold level being used to define the center of said region of interest, the width of said region of interest being defined by columns of said vertical intensity histogram on opposite sides of the column defining the center of said region of interest that have pixel intensities greater than said threshold level.

9. A touch system according to claim 8 wherein the digital signal processor of each digital camera analyses the pixels in said region of interest to locate the pixel row where the pointer tip is located and determine said pointer tip location z.

10. A touch system according to claim 9 wherein the digital signal processor of each digital camera creates a mask in said region of interest with white pixels representing said pointer and black pixels representing background to enable said median line x and pointer tip location to be calculated.

11. A touch system according to claim 10 wherein the digital signal processor of each digital camera further executes an update background image routine to update the background image after each image is acquired.

12. A touch system according to claim 11 wherein during said update background image routine, the digital signal processor of each digital camera uses the equation:

B_n+1(i,j)=(1-a)B_n(i,j)+aI(i,j)

where;

B_n+1 is the new background image;

B_n is the current background image;

I is the current acquired image;

i,j are the row and column coordinates of the background image pixels being updated; and

a is a number between 0 and 1 that indicates the degree of learning that should be taken from the current acquired image I.

13. A touch system according to claim 12 wherein the digital signal processor of each digital camera further determines the differences between each acquired image and the background image to detect changing light conditions.

14. A touch system according to claim 13 wherein each digital camera transmits light condition information to said processor, said processor using said light condition information to adjust the exposure of each said digital camera.

15. A touch system according to claim 14 wherein the digital signal processor of each digital camera further executes a create packet routine to package said image data and light condition information into said packets.

16. A touch system according to claim 3 wherein for image data received from each digital camera, said processor calculates an angle φ

_cam using the equation;

$\tan <br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{\varphi <br><br>}}_{\mathrm{cam}}=\frac{2<br><br>\left(\frac{x}{a}\right)<br><br>\tan <br><br>\frac{\mathrm{FOV}}{2}}{1-\left(2<br><br>\frac{x}{a}-1\right)<br><br>{\tan }^2<br><br>\frac{\mathrm{FOV}}{2}}$

where;

x is the number representing the median line or tip of the pointer; and

a is the total length enclosed by the field of view (FOV) of the digital camera at a distance from the camera;

said processor using the calculated angles to determine the position of the pointer relative to said touch surface using triangulation.

17. A touch system according to claim 16 wherein said calculated angles are adjusted to take into account digital camera offsets prior to determination of said pointer position.

18. A touch system according to claim 17 wherein said digital camera offsets are determined during execution of a digital camera calibration routine.

19. A touch system according to claim 17 including at least three digital cameras, said processor determining the pointer position using triangulation for pairs of digital cameras and averaging the determined pointer positions.

20. A touch system according to claim 19 wherein said processor further executes a touch surface determination routine to calculate the orientation of the touch surface as seen by each digital camera to determine when the pointer is in contact with said touch surface and when said pointer is hovering above said touch surface.

21. A touch system according to claim 20 wherein said processor further calculates the velocity of a pointer as said pointer moves toward said touch surface within the fields of view of said digital cameras.

22. A touch system according to claim 20 wherein said processor tracks said pointer within the fields of view of said digital cameras.

23. A touch system according to claim 21 wherein said processor tracks said pointer using at least one Kalman filter.

24. A touch system according to claim 19 further including a computer coupled to said processor, said computer receiving said pointer location from said processor.

36. A touch system according to claim 35 wherein the digital signal processor of each digital camera further executes a create packet routine to package said image data and light condition information into packets for transmission to said processor.

25. A camera-based touch system comprising:

a generally rectangular passive touch surface on which contacts are made using a pointer;

a digital camera mounted adjacent each corner of said touch surface, said digital cameras having overlapping fields of view encompassing said touch surface, said digital cameras acquiring images of said touch surface and generating image data that includes the median line x and pointer tip location z of a pointer when said pointer is captured in images acquired by said digital cameras; and

a processor receiving and processing image data generated by said digital cameras to determine the location of said pointer relative to said touch surface and whether said pointer is in contact with said touch surface.

26. A touch system according to claim 25 wherein each of said digital cameras includes a pixel array with selectable pixel rows, pixel intensities of pixels in said selectable pixel rows being used during generation of said image data.

27. A touch system according to claim 26 wherein pixel intensities of pixels in a region of interest within said selectable pixel rows are used during generation of said image data.

28. A touch system according to claim 27 wherein each of said digital cameras includes a CMOS image sensor and a digital signal processor, said digital signal processor receiving image output from said image sensor and executing a find pointer routine to determine if a pointer is in each image acquired by said digital camera, and if so the median line x of said pointer.

29. A touch system according to claim 28 wherein during said find pointer routine, said digital signal processor of each digital camera builds a vertical intensity histogram including columns of pixel intensities representing differences between an acquired image and a background image, the column of said vertical intensity histogram having the largest pixel intensity above a threshold level being used to define the center of said region of interest, the width of said region of interest being defined by columns of said vertical intensity histogram on opposite sides of the column defining the center of said region of interest that have pixel intensities greater than a threshold level.

30. A touch system according to claim 29 wherein the digital signal processor of each digital camera analyses the pixels in said region of interest to locate the pixel row where the pointer tip is located and determine said pointer tip location z.

31. A touch system according to claim 30 wherein the digital signal processor of each digital camera creates a mask in said region of interest with white pixels representing said pointer and black pixels representing background to enable said median line x and pointer tip location to be calculated.

32. A touch system according to claim 31 wherein the digital signal processor of each digital camera further executes an update background image routine to update the background image after each image is acquired.

33. A touch system according to claim 32 wherein during said update background image routine, the digital signal processor of each digital camera uses the equation:

B_n+1(i,j)=(1-a)B_n(i,j)+aI(i,j)

where;

B_n+1 is the new background image;

B_n is the current background image;

I is the current acquired image;

i,j are the row and column coordinates of the background image pixels being updated; and

a is a number between 0 and 1 that indicates the degree of learning that should be taken from the current acquired image I.

34. A touch system according to claim 33 wherein the digital signal processor of each digital camera further determines the differences between each acquired image and the background image to detect changing light conditions.

35. A touch system according to claim 34 wherein each digital camera transmits light condition information to said processor, said processor using said light condition information to adjust the exposure of each said digital camera.

37. A touch system according to claim 26 wherein for image data received from each digital camera, said processor calculates an angle φ

_cam using the equation;

$\tan <br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{\varphi <br><br>}}_{\mathrm{cam}}=\frac{2<br><br>\left(\frac{x}{a}\right)<br><br>\tan <br><br>\frac{\mathrm{FOV}}{2}}{1-\left(2<br><br>\frac{x}{a}-1\right)<br><br>{\tan }^2<br><br>\frac{\mathrm{FOV}}{2}}$

where;

x is the number representing the median line or tip of the pointer; and

a is the total length enclosed by the field of view (FOV) of the digital camera at a distance from the camera;

said processor using the calculated angles to determine the position of the pointer relative to said touch surface using triangulation.

38. A touch system according to claim 37 wherein said calculated angles are adjusted to take into account digital camera offsets prior to determination of said pointer position.

39. A touch system according to claim 37 wherein said processor further executes a touch surface determination routine to calculate the orientation of the touch surface as seen by each digital camera to determine when the pointer is in contact with said touch surface and when said pointer is hovering above said touch surface.

40. A touch system according to claim 39 wherein said processor further calculates the velocity of a pointer as said pointer moves toward said touch surface within the fields of view of said digital cameras.

41. A touch system according to claim 40 wherein said processor track said pointer within the fields of view of said digital cameras.

42. A touch system according to claim 41 wherein said processor tracks said pointer using at least one Kalman filter.

43. A touch system according to claim 26 wherein said touch surface is bordered by a frame and wherein each of said digital cameras is mounted on a frame assembly positioned at a corner of said touch surface that is coupled to a frame, each digital camera being oriented so that the field of view thereof looks downward and generally along the plane of said touch surface.

44. A touch system according to claim 26 further including a computer coupled to said processor, said computer receiving pointer location information from said processor and using said pointer location information to update an applications program executed thereby.

45. A touch system according to claim 44 wherein computer display information is presented on said touch surface.

46. A method of detecting the position of a pointer relative to a touch surface comprising the steps of:

acquiring images of said touch surface from different locations using cameras having overlapping fields of view and generating image data; and

processing said image data to detect the existence of a pointer within said acquired images and to determine the location of said pointer relative to said touch surface.

47. The method of claim 46 wherein during the processing step, the location of said pointer relative to said touch screen is determined using triangulation.

48. The method of claim 47 wherein during said processing step, the images are processed to determine when said pointer is in contact with said touch surface and when said pointer is hovering above said touch surface.

49. The method of claim 48 wherein said processing step further includes the step of tracking the pointer as the pointer approaches the touch surface.