Method and apparatus for calibrating projector-camera system

• US 6,618,076 B1
• Filed: 11/14/2000
• Issued: 09/09/2003
• Est. Priority Date: 12/23/1999
• Status: Expired due to Fees

• Current Assignee: Justsystems Corporation
• Original Assignee: Justsystems Corporation
• Inventors: Sukthankar, Rahul, Mullin, Matthew, Stockton, Robert
• Primary Examiner(s): Miller, John
• Assistant Examiner(s): NATNAEL, PAULOS M
• Application Number:
  US09/712,747
• Time in Patent Office: 1,029 Days
• Field of Search: 348/745, 348/746, 348/747, 348/180-1, 348/187, 348/189, 348/191, 348/806, 348/807, 348/658, 348/175-178, 348/744, 315/368.11, 315/368.13, 315/368.22, 353/122, 345/156, 345/158
• US Class Current: 348/180
• CPC Class Codes: H04N 17/04 for receivers
• CPC Class Codes: H04N 5/74 Projection arrangements for image reproduction, e.g. using eidophor
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

The present invention enables a user to automatically calibrate a projector-camera system to recover the mapping from a given point in the source (pre-projection) image and its corresponding point in the camera image, and vice versa. One or more calibration patterns are projected onto a flat surface with possibly unknown location and orientation by a projector with possibly unknown location, orientation and focal length. Images of these patterns are captured by a camera mounted at a possibly unknown location, orientation and with possibly unknown focal length. Parameters for mapping between the source image and the camera image are computed. The present invention can become an essential component of a projector-camera system, such as automatic keystone correction and vision-based control of computer systems.

Claims

1. A computer implemented method for automatically calibrating a system having a camera with a camera image frame, a projector with a source image frame and a projection surface with a projection image frame, comprising the steps of:

arbitrarily placing the camera and the projector;

intersecting a camera field of view of the camera with a projector field of projection of the projector on the projection surface;

projecting an image of a calibration pattern with a plurality of calibration pattern feature points onto the projection surface;

capturing the image of the calibration pattern by the camera;

identifying locations of calibration pattern feature points in the camera image frame of the captured image using a feature extraction algorithm; and

obtaining parameters for mapping utilizing techniques of linear algebra, given a location of a calibration pattern feature point in the source image frame and a corresponding location of the calibration pattern feature point in the camera image frame such that the system is calibrated,

wherein the location of the calibration pattern feature point, i, in the camera image frame is (X_i, Y_i), the corresponding location of the calibration pattern feature point in the source image frame is (x_i,y_i), $\begin{array}{c}{A}_i=\left(\begin{array}{ccccccccc}{X}_i& Y_i& 1& 0& 0& 0& -X_i<br><br>x_i& -Y_i<br><br>x_i& -x_i\\ 0& 0& 0& X_i& Y_i& 1& -X_i<br><br>y_i& -Y_i<br><br>y_i& -y_i\end{array}\right)\\ B=\underset{i}{\sum <br><br>}<br><br>A_i^{\mathrm{\tau <br><br>}}<br><br>A_i\end{array}$

the parameters of the mapping are represented by {overscore (p)}=(p₁. . . p₉), {overscore (p)} is a unit vector resulting in eight degrees of freedom, the {overscore (p)} that best maps the calibration pattern feature point from the camera image frame to the source image frame is given by the eigenvector corresponding to the smallest eigenvalue of the matrix B, and given the mapping {overscore (p)}, any given point (X, Y) in the camera image frame is transformable to its corresponding point (x,y) in the source image frame by the following equation, $\left(x,y\right)=\left(\frac{p_1<br><br>X+p_2<br><br>Y+p_3}{p_7<br><br>X+p_8<br><br>Y+p_9},\frac{p_4<br><br>X+p_5<br><br>Y+p_6}{p_7<br><br>X+p_8<br><br>Y+p_9}\right).$

2. The method of claim 1 wherein the projection surface is planar.

3. The method of claim 1 wherein a plurality of images of calibration patterns is projected onto the projection surface.

4. The method of claim 1 wherein the calibration pattern comprises a set of images, with each image formed as a single pixel on a contrasting background.

5. The method of claim 1 wherein the feature extraction algorithm includes the step of:

thresholding by intensity the pixel to create a binary image such that a centroid of the pixel is the location of a corresponding calibration pattern feature point.

6. The method of claim 1 wherein the calibration pattern includes four differently colored spots on a black background, with each colored spot located near a corner of the source image frame.

7. The method of claim 1 wherein the calibration pattern includes a set of images, with each image formed as a differently sized white rectangle on a black background.

8. The method of claim 7 further including the step of:

computing intersections of lines of corners of each differently sized white rectangle.

9. The method of claim 1 wherein the feature extraction algorithm includes the steps of:

thresholding by intensity the pixel to create a binary image;

utilizing a connected components algorithm to identify the largest region of bright pixels in the image corresponding to the differently sized white rectangle in the calibration pattern;

extracting the edges of the largest region; and

computing the intersections between adjacent edges, yielding the locations of the corners to sub-pixel accuracy.

10. The method of claim 1 wherein the location of the calibration pattern feature points is manually adjustable by a user.

11. The method of claim 1 wherein inverse mapping from the source image frame to the camera image frame exchanges (x_i,y_i) and (X_i, Y_j) in all cases.

12. The method of claim 1 wherein the projection surface includes a plurality of planar patches, a calibration pattern having at least four calibration pattern feature points is projected onto each of the planar patches, and parameters for mapping each planar patch is independently derived.

13. The method of claim 1 wherein the parameters for mapping are restricted between calibration pattern feature points in the camera image frame and the corresponding calibration pattern feature points in the projection image frame.

14. An automatic calibration system comprising:

a camera with a camera image frame and a camera field of view;

a projector with a source image frame and a projector field of projection;

a projection surface with a projection image frame;

a means for performing a feature extraction algorithm; and

a means for mapping parameters utilizing techniques of linear algebra,

wherein the camera and the projector are arbitrarily placed, a camera field of view of the camera intersects a projector field of projection of the projector on the projection surface, the projector projects an image of a calibration pattern with a plurality of calibration pattern feature points onto the projection surface, the camera captures the image of the calibration pattern, the locations of calibration pattern feature points in the camera image frame of the captured image are identified using the feature extraction algorithm, and the parameters for mapping are obtained, given a location of a calibration pattern feature point in the source image frame and a corresponding location of the calibration pattern feature point in the camera image frame;

wherein the location of the calibration pattern feature point, i, in the camera image frame is (X_i, Y_i), the corresponding location of the calibration pattern feature point in the source image frame is (x_i,y_i), $\begin{array}{c}{A}_i=\left(\begin{array}{ccccccccc}{X}_i& Y_i& 1& 0& 0& 0& -X_i<br><br>x_i& -Y_i<br><br>x_i& -x_i\\ 0& 0& 0& X_i& Y_i& 1& -X_i<br><br>y_i& -Y_i<br><br>y_i& -y_i\end{array}\right)\\ B=\underset{i}{\sum <br><br>}<br><br>A_i^{\mathrm{\tau <br><br>}}<br><br>A_i\end{array}$

the parameters of the mapping are represented by {overscore (p)}=(p₁. . . p₉), {overscore (p)} is a unit vector resulting in eight degrees of freedom, the {overscore (p)} that best maps the calibration pattern feature point from the camera image frame to the source image frame is given by the eigenvector corresponding to the smallest eigenvalue of the matrix B, and given the mapping {overscore (p)}, any given point (X, Y) in the camera image frame is transformable to its corresponding point (x,y) in the source image frame by the following equation, $\left(x,y\right)=\left(\frac{p_1<br><br>X+p_2<br><br>Y+p_3}{p_7<br><br>X+p_8<br><br>Y+p_9},\frac{p_4<br><br>X+p_5<br><br>Y+p_6}{p_7<br><br>X+p_8<br><br>Y+p_9}\right).$

15. The automatic calibration system of claim 14 wherein at least four calibration pattern feature points are identified in the camera image frame of the captured image.

16. The automatic calibration system of claim 14 wherein the projection surface is planar.

17. The automatic calibration system of claim 14 wherein the projection surface is piece-wise planar.

18. The automatic calibration system of claim 14 wherein the camera is a digital camera.

19. The automatic calibration system of claim 14 wherein the projector is an LCD projector.

20. The automatic calibration system of claim 14 further comprising a computer in communication with the projector, wherein the camera is in communication with the computer.

21. The automatic calibration system of claim 14 wherein a plurality of images of calibration patterns is projected onto the projection surface.

22. The automatic calibration system of claim 14 wherein the calibration pattern comprises a set of images, with each image formed as a single pixel on a contrasting background.

23. The automatic calibration system of claim 14 wherein the feature extraction algorithm creates a binary image by thresholding by intensity a pixel such that a centroid of the pixel is the location of a corresponding calibration pattern feature point.

24. The automatic calibration system of claim 14 wherein the calibration pattern includes four differently colored spots on a black background, with each colored spot located near a corner of the source image frame.

25. The automatic calibration system of claim 14 wherein the calibration pattern includes a set of images, with each image formed as a differently sized white rectangle on a black background.

26. The automatic calibration system of claim 25 wherein intersections of lines of corners of each differently sized white rectangle are computed.

27. The automatic calibration system of claim 14 wherein the feature extraction algorithm is performed by thresholding by intensity the pixel to create a binary image, utilizing a connected components algorithm to identify the largest region of bright pixels in the image corresponding to the differently sized white rectangle in the calibration pattern, extracting the edges of the largest region, and computing the intersections between adjacent edges, yielding the locations of the corners to sub-pixel accuracy.

28. The automatic calibration system of claim 14 wherein the calibration pattern is selected from the group consisting of grids, lines, polygons and dots.

29. The automatic calibration system of claim 14 wherein the accuracy of the calibration pattern is maximized using statistical error models.

30. The automatic calibration system of claim 14 wherein the calibration pattern feature points are in a maximally spaced apart relationship.

31. The automatic calibration system of claim 14 wherein the projected calibration pattern is non-visible light.

32. The automatic calibration system of claim 14 wherein the non-visible light is infrared light.

33. The automatic calibration system of claim 14 wherein the location of the calibration pattern feature points is manually adjustable by a user.

34. The automatic calibration system of claim 14 wherein inverse mapping from the source image frame to the camera image frame exchanges (x_i,y_i) and (X_i, Y_i) in all cases.

35. The automatic calibration system of claim 14 wherein the projection surface includes a plurality of planar patches, a calibration pattern having at least four calibration pattern feature points is projected onto each of the planar patches, and the parameters for mapping each planar patch is independently derived.

36. The automatic calibration system of claim 14 wherein the captured image emanates from a computer CRT monitor.

37. The automatic calibration system of claim 14 wherein the captured image emanates from an LCD display device.

38. The automatic calibration system of claim 14 wherein the system periodically repeats the steps of identifying the locations of calibration pattern feature points in the camera image frame of the captured image using a feature extraction algorithm, and obtaining parameters for mapping utilizing the techniques of linear algebra, given a location of a calibration pattern feature point in the source image frame and a corresponding location of the calibration pattern feature point in the camera image frame.

39. The automatic calibration system of claim 14 wherein the parameters for mapping are restricted between calibration pattern feature points in the camera image frame and the corresponding calibration pattern feature points in the projection image frame.

40. The automatic calibration system of claim 14 wherein different regions of the camera image frame are independently calibrated.

41. The automatic calibration system of claim 14 wherein different regions of the source image frame are independently calibrated.

42. The automatic calibration system of claim 14 wherein the calibration pattern is visible to a user.

43. The automatic calibration system of claim 14 wherein the calibration pattern is detectable by the camera.

44. The automatic calibration system of claim 14 wherein the type of calibration pattern utilized is determined adaptively during the calibration process.

45. The automatic calibration system of claim 14 wherein the means for performing the feature extraction algorithm is a computer.

46. The automatic calibration system of claim 14 wherein the means for mapping the parameters utilizing techniques of linear algebra is a computer.