Calibrating camera offsets to facilitate object position determination using triangulation

• US 6,919,880 B2
• Filed: 06/01/2001
• Issued: 07/19/2005
• Est. Priority Date: 06/01/2001
• Status: Active Grant

• Current Assignee: Smart Technologies ULC (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Smart Technologies Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Morrison, Gerald, Holmgren, David
• Primary Examiner(s): Hjerpe, Richard
• Assistant Examiner(s): NGUYEN, KIMNHUNG T
• Application Number:
  US09/870,698
• Publication Number:
  US 20030043116A1
• Time in Patent Office: 1,509 Days
• Field of Search: 345/173, 345/174, 345175-180, 178/18.01, 178/18.03, 178/18.04
• US Class Current: 345/173
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421

Abstract

A touch system includes a reference frame, and at least two cameras having fields of view that overlap within the reference frame. The position of an object relative to the reference frame is determined from captured images of the object based on triangulation. The fields of view of the at least two cameras are rotated with respect to the coordinate system of the reference frame to define offset angles. The touch system is calibrated by: capturing an image of the object using each the at least two cameras at least one location within the reference frame; and for each location: determining the position of the object within each image, the position of the object within each image being represented by an angle φ, the angle being equal to the angle formed between an extremity of the field of view extending beyond the reference frame and a line extending from the camera that intersects the object within the image; and mathematically calculating the offset angles of the at least two cameras based on the angle determined for each image and the position of the at least two cameras relative to the coordinate system assigned to the reference frame.

Claims

1. A method of determining the position of an object relative to a rectangular reference frame from captured images of the object based on multiple triangulation results, the captured images being taken by at least two pair of cameras at the corners of said reference frame having fields of view encompassing said reference frame, each of said cameras having an offset angle resulting in an extremity of the field of view thereof extending beyond a boundary of said reference frame, said method comprising the steps of:

capturing an image of the object using each camera of said at least two pair at at least one location within said reference frame;

for each location;

determining the position of the object within each captured image and for each captured image placing the determined position into a coordinate system corresponding to that of said reference frame, wherein the determined position of the object within each image is represented by an angle φ

, said angle being equal to the angle formed between the extremity of the field of view extending beyond the reference frame boundary and a line extending from the camera that intersects the object within the image; and

processing the determined positions to determine the position of the object at each location and the offset angle of said at least one camera, wherein during said processing each said angle φ

is converted to an angle ω

, said angle ω

being represented by;

ω

=α

−

δ

where;

δ

is the camera offset angle; and

α

is equal to the angle φ

with the camera offset angle removed and referenced to the y-axis of the reference frame coordinate system and wherein each said angle ω

is fitted to the equation;

${\mathrm{\omega <br><br>}}_{\mathrm{cam}}=\arctan <br><br>\left[\frac{x_{\mathrm{cam}}-x_i}{y_{\mathrm{cam}}-y_i}\right]+{\mathrm{\delta <br><br>}}_{\mathrm{cam}}$

where;

x_cam and y_cam are the rectangular coordinates of the camera; and

x_i and y_i are the rectangular coordinates of the object, thereby to yield the rectangular position (x_i, y_i) and the camera offset angle.

2. The method of claim 1 wherein the fitting is performed using a linearization technique.

3. The method of claim 2 wherein said linearization technique is the Moore-Penrose pseudo-inverse method.

4. A method of determining the position of an object relative to a reference frame from captured images of the object based on multiple triangulation results, the captured images being taken by at least two pair of cameras having fields of view encompassing the reference frame, an extremity of the field of view of each camera encompassing a boundary of said reference frame, at least one of said cameras being offset causing the extremity of the field of view thereof to extend beyond said boundary, the offset defining an offset angle, said method comprising the steps of:

determining the position of the object within each image, the position of the object within each image being represented by an angle, said angle being equal to the angle formed between the extremity of the field of view of the camera that acquired the image and a line extending from that camera that intersects the object within the image;

determining the offset angle for each offset camera;

for each offset camera subtracting the offset angle from the angle representing the position of the object within the image taken by said offset camera to calibrate the angle; and

for each pair of cameras using the calibrated angles to calculate the position of the object with respect to the reference frame using triangulation.

5. The method of claim 4 wherein two or more of said cameras are offset.

10. The method of claim 5 wherein the offset angle for each offset camera is determined using a least squares method.

11. The method of claim 5 wherein said at least two pair of cameras include at least three cameras, each camera being located adjacent a different corner of said reference frame.

6. In a touch system including at least two pair of cameras and a processor to process images acquired by said at least two pair cameras, where the position of an object that is within the fields of view of said cameras relative to a reference frame is determined by triangulating object position data in images acquired by the cameras of each pair, a method of calibrating the touch system comprising the steps of:

determining an offset angle of each camera relative to the reference frame, said offset angle representing the degree by which the field of view of the camera extends beyond said reference frame;

for each camera, using the offset angle to calibrate the object position data developed from the image acquired by that camera; and

using the calibrated object position data during triangulation for each pair of cameras to determine the position of said object relative to said reference frame.

12. The method of claim 6 wherein the offset angle for each offset camera is determined using a least squares method.

13. The method of claim 6 wherein said at least two pair of cameras include at least three cameras, each camera being located adjacent a different corner of said reference frame.

7. In a touch system including a reference frame, and at least two pair of cameras having fields of view that encompass said reference frame, wherein the position of an object relative to the reference frame is determined from captured images of the object based on multiple triangulation results, and wherein the fields of view of at least some of said cameras are rotated with respect to the coordinate system of said reference frame to define offset angles, a method of calibrating said touch system comprising the steps of:

capturing an image of the object using each camera of said at least two pair at at least one location within said reference frame; and

for each location;

determining the position of the object within each captured image, the position of the object within each captured image being represented by an angle φ

, said angle being equal to the angle formed between an extremity of the field of view of the camera that acquired the image extending beyond the reference frame and a line extending from that camera that intersects the object within the image; and

mathematically calculating the offset angles of the cameras having rotated fields of view based on the angle determined for each image and the position of the cameras relative to the coordinate system assigned to said reference frame.

8. The method of claim 7 wherein the offset angle of each camera is calculated using a least squares method.

14. The method of claim 7 wherein during said processing each said angle is converted into a rectangular (x_i,y_i) position within the reference frame coordinate system.

15. The method of claim 14 wherein captured images are acquired by cameras at the corners of a rectangular reference frame, each of said cameras having a field of view offset with respect to said reference frame, during said processing each said angle φ

being converted to an angle ω

, said angle ω

being represented by;

ω

=α

−

δ

where;

δ

is the camera offset angle; and

α

is equal to the angle φ

with the camera offset angle removed and referenced to the y-axis of the reference frame coordinate system and wherein each said angle ω

is fitted to the equation;

${\mathrm{\omega <br><br>}}_{\mathrm{cam}}=\arctan <br><br>\left[\frac{x_{\mathrm{cam}}-x_i}{y_{\mathrm{cam}}-y_i}\right]+{\mathrm{\delta <br><br>}}_{\mathrm{cam}}$

where;

x_cam and y_cam are the rectangular coordinates of the camera; and

x_i and y_i are the rectangular coordinates of the object, thereby to yield the rectangular position (x_i, y_i) and the camera offset angle.

16. The method of claim 15 wherein the fitting is performed using a linearization technique.

17. The method of claim 15 wherein said linearization technique is the Moore-Penrose pseudo-inverse method.

9. A touch system comprising:

a generally rectangular reference frame surrounding a touch surface, one corner of the reference frame defining the origin of a coordinate system assigned to said touch surface;

a camera adjacent each corner of the reference frame, each camera being aimed towards said touch surface and capturing images of said touch surface within the field of view thereof, fields of view of said cameras overlapping within said reference frame, the fields of view of said cameras being offset with respect to said reference frame; and

a processor processing the captured images and generating object position data when an object appears in images, said processor determining the position of said object relative to said origin in rectangular coordinates using said object position data based on multiple triangulation results, wherein said processor further executes a calibration routine to determine offset angles of said cameras, said offset angles being used by said processor to adjust said object position data thereby to align said multiple triangulation results prior to said position determination.

18. A touch system according to claim 9 wherein said processor determines the offset angles using a least squares method.

19. A touch system comprising:

a substantially rectangular touch surface;

imaging devices mounted adjacent at least three corners of said touch surface to define at least two triangulation pair of imaging devices, each imaging device having a field of view looking across said touch surface, said imaging devices being oriented to capture overlapping images of said touch surface; and

at least one processing device processing captured images to determine the position of at least one pointer appearing in the captured images based on multiple triangular results, the fields of view of said imaging devices being calibrated by said at least one processing device to determine offset angles of said imaging devices prior to determining the position of the at least one pointer thereby to align said multiple triangulation results.

20. A touch system according to claim 19 comprising an imaging device at each corner of said touch surface to define at least four triangulation pair of imaging devices.

21. A touch system according to claim 19 wherein during said calibration, said at least one processing device determines the degree by which the field of view of each imaging device extends beyond the boundary of said touch surface.

22. A touch system according to claim 21 wherein the degree by which the field of view of each imaging device extends beyond the boundary of said touch surface is determined using a least squares method.

27. A touch system according to claim 21 wherein the pointer in each image is represented by an angle φ

, each angle φ

being equal to the angle formed between the extremity of the field of view extending beyond the touch surface and a line extending from the imaging device that intersects the pointer and wherein said at least one processing device converts each said angle φ

to an angle ω

, said angle ω

being represented by;

ω

=α

−

δ

where;

δ

is an imaging device offset angle representing the degree by which the field of view of the imaging device extends beyond said boundary; and

α

is equal to the angle φ

with the imaging device offset angle removed and referenced to the y-axis of the touch surface coordinate system and wherein each said angle ω

is fitted to the equation;

${\mathrm{\omega <br><br>}}_{\mathrm{cam}}=\arctan <br><br>\left[\frac{x_{\mathrm{cam}}-x_i}{y_{\mathrm{cam}}-y_i}\right]+{\mathrm{\delta <br><br>}}_{\mathrm{cam}}$

where;

x_cam and y_cam are the rectangular coordinates of the imaging device; and

x_i, and y_i are the rectangular coordinates of the pointer, thereby to yield the rectangular position (x_i, y_i) and the imaging device offset angle.

28. A touch system according to claim 27 wherein said at least one processing device uses a linerization technique to fit each said angle ω

to the equation.

23. A user input system comprising:

at least two pair of imaging devices having overlapping fields of view oriented to capture images of a region of interest in which at least one pointer can be positioned; and

at least one processing device processing pointer data extracted from the captured images acquired by the imaging devices using triangulation to yield a triangulation result for each pair of imaging devices thereby to determine the position of said at least one pointer within said region of interest, said at least one processing device adjusting the pointer data prior to processing by determining offset angles of said imaging devices to compensate for fields of view of said imaging devices that extend beyond the periphery of said region of interest thereby to align the triangulation results.

24. A user input system according to claim 23 wherein the degree by which the field of view of each imaging device extends beyond the boundary of said touch surface is determined using a least squares method.

25. A user input system according to claim 23 wherein said region of interest encompasses a touch surface.

26. A user input system according to claim 25 wherein said at least two pair of imaging devices include at least three imaging devices, each being located adjacent a different corner of said touch surface.

29. A user input system according to claim 25 wherein the pointer in each image is represented by an angle φ

, each angle φ

being equal to the angle formed between the extremity of the field of view extending beyond the touch surface and a line extending from the imaging device that intersects the pointer and wherein said at least one processing device converts each said angle φ

to an angle ω

, said angle ω

being represented by;

ω

=α

−

δ

where;

δ

is an imaging device offset angle representing the degree by which the field of view of the imaging device extends beyond said boundary; and

α

is equal to the angle φ

with the imaging device offset angle removed and referenced to the y-axis of the touch surface coordinate system and wherein each said angle ω

is fitted to the equation;

${\mathrm{\omega <br><br>}}_{\mathrm{cam}}=\arctan <br><br>\left[\frac{x_{\mathrm{cam}}-x_i}{y_{\mathrm{cam}}-y_i}\right]+{\mathrm{\delta <br><br>}}_{\mathrm{cam}}$

where;

x_cam and y_cam are the rectangular coordinates of the imaging device; and

x_i and y_i are the rectangular coordinates of the pointer, thereby to yield the rectangular position (x_i, y_i) and the imaging device offset angle.

30. A user input system according to claim 29 wherein said at least one processing device uses a linerization technique to fit each said angle ω

to the equation.

31. A user input system according to claim 25 comprising an imaging device at each corner of said touch surface to define at least four triangulation pair of imaging devices.

32. A user input system according to claim 31 wherein the pointer in each image is represented by an angle φ

, each angle φ

being equal to the angle formed between the extremity of the field of view extending beyond the touch surface and a line extending from the imaging device that intersects the pointer and wherein said at least one processing device converts each said angle φ

to an angle ω

, said angle ω

being represented by;

ω

=α

−

δ

where;

δ

is an imaging device offset angle representing the degree by which the field of view of the imaging device extends beyond said boundary; and

α

is equal to the angle φ

with the imaging device offset angle removed and referenced to the y-axis of the touch surface coordinate system and wherein each said angle ω

is fitted to the equation;

${\mathrm{\omega <br><br>}}_{\mathrm{cam}}=\arctan <br><br>\left[\frac{x_{\mathrm{cam}}-x_i}{y_{\mathrm{cam}}-y_i}\right]+{\mathrm{\delta <br><br>}}_{\mathrm{cam}}$

where;

x_cam and y_cam are the rectangular coordinates of the imaging device; and

x_i and y_i are the rectangular coordinates of the pointer, thereby to yield the rectangular position (x_i, y_i) and the imaging device offset angle.

33. A user input system according to claim 32 wherein said at least one processing device uses a linerization technique to fit each said angle ω

to the equation.