Apparatus and method for detecting a pointer relative to a touch surface

• US 20050248539A1
• Filed: 05/05/2004
• Published: 11/10/2005
• Est. Priority Date: 05/05/2004
• Status: Active Grant

• Current Assignee: PixArt Imaging Incorporated
• Original Assignee: Smart Technologies ULC (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Holmgren, David E., Morrison, Gerald D.
• Granted Patent: US 7,492,357 B2
• Time in Patent Office: Days
• US Class Current: 345/173
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/85 Stereo camera calibration

Abstract

An apparatus for detecting a pointer relative to a touch surface includes at least two spaced imaging assemblies having overlapping fields of view encompassing the touch surface. The imaging assemblies see the touch surface in three-dimensions as a perspective view. The imaging assemblies acquire overlapping images from different locations. A processor receives and processes image data generated by at least one of the imaging assemblies to determine the location of the pointer relative to the touch surface.

Claims

1. An apparatus for detecting a pointer relative to a touch surface comprising:

at least two spaced imaging devices having overlapping fields of view encompassing said touch surface, said imaging devices seeing said touch surface in three-dimensions as a perspective view, said imaging devices acquiring overlapping images from different locations; and

a processor receiving and processing image data generated by at least one of said imaging devices to determine the location of the pointer relative to said touch surface.

2. An apparatus according to claim 1 wherein each imaging device is calibrated to establish the relationship between points (X,Y,Z) in its perspective view and points (x,y) in acquired images, each imaging device generating pointer co-ordinate data when a pointer is captured in an acquired image.

3. An apparatus according to claim 2 wherein said processor triangulates the pointer co-ordinate data to determine the location of the pointer relative to said touch surface.

4. An apparatus according to claim 3 including a pair of imaging devices.

5. An apparatus according to claim 4 wherein each imaging device is positioned adjacent a different corner of said touch surface.

6. An apparatus according to claim 5 wherein each imaging device is spaced from and spaced in front of said touch surface.

7. An apparatus according to claim 6 wherein each imaging device is positioned relative to said touch surface so that at a minimum the entire periphery of the touch surface is within its field of view.

8. An apparatus according to claim 7 wherein each imaging device is spaced in front of said touch surface by a distance equal to at least 2.5 cm.

9. An apparatus according to claim 8 wherein said touch surface is bordered by a bezel.

10. An apparatus according to claim 7 wherein each imaging device is aimed so that the optical axis thereof is in line with a corner of said touch surface and forms an angle with the top edge of said touch surface equal to approximately 45 degrees.

11. An apparatus according to claim 3 wherein during calibration external orientation parameters (X₀,Y₀,Z₀) and (ω

,φ

,κ

) of each imaging device are determined where;

(X₀,Y₀,Z₀) is the spatial location of the optical center of the imaging device; and

(ω

,φ

,κ

) are the orientation angles of the optical axis of the imaging device with respect to the three-dimensional co-ordinate system of the touch surface.

12. An apparatus according to claim 11 wherein during calibration internal orientation parameters ƒ

,x₀,y₀ and Δ

_x,Δ

_y of each imaging device are also determined where;

(Δ

_x,Δ

_y) represent distortion terms introduced due to the imperfect nature of a lens of the imaging device;

ƒ

is the focal length of the imaging device; and

(x₀,y₀) are the co-ordinates of the principal point of the imaging device.

13. An apparatus according to claim 12 wherein said relationship is expressed using collinearity equations as:

$x=x_0-{\mathrm{\Delta <br><br>}}_x-\frac{{\mathrm{fR}}_1}{R_3}$ $y=y_0-{\mathrm{\Delta <br><br>}}_y-\frac{{\mathrm{fR}}_2}{R_3}$

where;

(x,y) are the co-ordinates of a point in a captured image corresponding to a point (X,Y,Z) in the three-dimensional perspective view; and

(R₁,R₂,R₃) are terms depending on point (X,Y,Z), the spatial location (X₀,Y₀,Z₀) and the orientation angles (ω

,φ

,κ

).

14. An apparatus according to claim 13 wherein during calibration, calibration points (X,Y,Z) on said touch surface and image points (x,y) corresponding to said calibration points are measured, said collinearity equations being solved using said measured calibration and image points thereby to determine said external and internal orientation parameters.

15. An apparatus according to claim 14 wherein said collinearity equations are solved using a least-squares method.

16. An apparatus according to claim 14 wherein said calibration points are at spaced locations along the periphery of said touch surface.

17. An apparatus according to claim 16 wherein said calibration points are located at the corners and edge mid-points of said touch surface.

18. An apparatus according to claim 11 wherein said external orientation parameters are determined using a vanishing point method.

19. An apparatus according to claim 18 wherein the determined external orientation parameters are refined using a least-squares method.

20. An apparatus according to claim 18 wherein said external and internal orientation parameters are determined using planar homography.

21. An apparatus according to claim 18 wherein said external orientation parameters are determined using a three point method.

22. An apparatus according to claim 3 wherein each imaging device also generates a certainty value representing the degree of certainty that the imaging device has positively identified the pointer in the acquired image.

23. An apparatus according to claim 22 wherein said certainty value is used by said processor to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said touch surface.

24. An apparatus according to claim 23 wherein said processor ignores pointer co-ordinate data generated by said imaging device when the certainty value associated therewith is below a threshold level.

25. An apparatus according to claim 24 including a pair of imaging devices.

26. An apparatus according to claim 25 wherein each imaging device is positioned adjacent a different corner of said touch surface.

27. An apparatus according to claim 26 wherein each imaging device is spaced from and spaced in front of said touch surface.

28. An apparatus according to claim 27 wherein each imaging device is positioned relative to said touch surface so that at a minimum the entire periphery of the touch surface is within its field of view.

29. An apparatus according to claim 28 wherein each imaging device is aimed so that the optical axis thereof is in line with the corner of said touch surface and forms an angle with the top edge of said touch surface equal to approximately 45 degrees.

30. An apparatus according to claim 3 wherein said imaging devices communicate to assist in determining a pointer in acquired images.

31. An apparatus according to claim 30 wherein the imaging device that detects a pointer in its acquired image first communicates data to the other imaging device to assist that imaging device to detect the pointer in its acquired image.

32. An apparatus according to claim 31 wherein each imaging device also generates a certainty value representing the degree of certainty that the imaging device has positively identified the pointer in the acquired image.

33. An apparatus according to claim 32 wherein said certainty value is used by said processor to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said touch surface.

34. An apparatus according to claim 33 wherein said processor ignores pointer co-ordinate data generated by said imaging device when the certainty value associated therewith is below a threshold level.

35. An apparatus according to claim 34 including a pair of imaging devices.

36. An apparatus according to claim 35 wherein each imaging device is positioned adjacent a different corner of said touch surface.

37. An apparatus according to claim 36 wherein each imaging device is spaced from and spaced in front of said touch surface.

38. An apparatus according to claim 37 wherein each imaging device is positioned relative to said touch surface so that at a minimum the entire periphery of the touch surface is within its field of view.

39. An apparatus according to claim 38 wherein each imaging device is aimed so that the optical axis thereof is in line with the corner of said touch surface and forms an angle with the top edge of said touch surface equal to approximately 45 degrees.

40. An apparatus according to claim 3 wherein each imaging device processes a subset of pixels in each acquired image.

41. An apparatus according to claim 40 wherein said pixel subset corresponds to the boundary of said touch surface within the perspective view of said imaging device.

42. An apparatus according to claim 41 wherein each imaging device also generates a certainty value representing the degree of certainty that the imaging device has positively identified the pointer in the acquired image.

43. An apparatus according to claim 42 wherein said certainty value is used by said processor to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said touch surface.

44. An apparatus according to claim 43 wherein said processor ignores pointer co-ordinate data generated by said imaging device when the certainty value associated therewith is below a threshold level.

45. An apparatus according to claim 44 including a pair of imaging devices.

46. An apparatus according to claim 45 wherein each imaging device is positioned adjacent a different corner of said touch surface.

47. An apparatus according to claim 46 wherein each imaging device is spaced from and spaced in front of said touch surface.

48. An apparatus according to claim 47 wherein each imaging device is positioned relative to said touch surface so that at a minimum the entire periphery of the touch surface is within its field of view.

49. An apparatus according to claim 48 wherein each imaging device is aimed so that the optical axis thereof is in line with the corner of said touch surface and forms an angle with the top edge of said touch surface equal to approximately 45 degrees.

50. An apparatus according to claim 4 wherein said imaging devices are portable.

51. An apparatus according to claim 50 wherein each imaging device includes a digital camera and a digital signal processor mounted within a housing, said digital signal processor processing image frames acquired by said digital camera to generate said pointer co-ordinate data.

52. An apparatus according to claim 50 wherein each imaging device includes a digital camera and a digital signal processor, both imaging devices being mounted within a single housing, said digital signal processor processing image data acquired by said digital camera to generate said pointer co-ordinate data.

53. An apparatus according to claim 3 including three or more imaging devices at spaced locations along said touch surface, each imaging device having a field of view encompassing a different portion of said touch surface.

54. An apparatus according to claim 53 wherein said imaging devices are arranged in pair, with each pair of imaging devices viewing a different portion of said touch surface.

55. A camera-based touch system comprising:

a generally rectangular passive touch surface on which contacts are made using a pointer;

camera devices removably mounted adjacent at least two corner of said touch surface, each of said camera devices having a field of view looking across and back towards said touch surface and being disposed in front of the plane of the touch surface, the fields of view of said camera devices overlapping over said touch surface, said camera devices acquiring images of said touch surface; and

a processor receiving and processing said image data to determine the location of said pointer relative to said touch surface via triangulation.

56. A touch system according to claim 55 wherein each camera device is calibrated to establish the relationship between points (X,Y,Z) in its perspective view and points (x,y) in acquired images, each camera device generating pointer co-ordinate data when a pointer is captured in an acquired image.

57. A touch system according to claim 56 wherein each camera device processes a subset of pixels in each acquired image.

58. A touch system according to claim 57 wherein said pixel subset corresponds to the boundary of said touch surface within the perspective view of said camera device.

59. A touch system according to claim 58 wherein each camera device is spaced in front of said touch surface a sufficient distance so that at a minimum each camera device sees the four corners of said touch surface with its field of view.

60. A touch system according to claim 59 wherein each camera device also generates a certainty value representing the degree of certainty that the camera device has positively identified the pointer in the acquired image.

61. A touch system according to claim 60 wherein said certainty value is used by said processor to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said touch surface.

62. A touch system according to claim 61 wherein said processor ignores pointer co-ordinate data generated by said camera device when the certainty value associated therewith is below a threshold level.

63. An apparatus for detecting a pointer relative to a generally rectangular touch surface comprising:

at least two spaced imaging devices having overlapping fields of view encompassing said touch surface, said imaging devices seeing said touch surface in three-dimensions as a perspective view with the perspective view including at least the four corner of said touch surface, said imaging devices acquiring overlapping images from different locations; and

a processor receiving and processing image data generated by at least one of said imaging devices to determine the location of the pointer relative to said touch surface using triangulation.

64. An apparatus according to claim 63 wherein each imaging device is calibrated to establish the relationship between points (X,Y,Z) in its perspective view and points (x,y) in acquired images, each imaging device outputting pointer co-ordinate data when a pointer is captured in an acquired image.

65. An apparatus according to claim 64 wherein each imaging device is spaced in front of said touch surface a sufficient distance to inhibit its view of the entire touch surface from being obstructed.

66. An apparatus according to claim 65 wherein each imaging device also generates a certainty value representing the degree of certainty that the imaging device has positively identified the pointer in the acquired image.

67. An apparatus according to claim 66 wherein said certainty value is used by said processor to determine pointer co-ordinate data to be used to determine the position of said pointer relative to said touch surface.

68. An apparatus according to claim 67 wherein said processor ignores pointer co-ordinate data generated by said imaging device when the certainty value associated therewith is below a threshold level.