Real-time geometry aware projection and fast re-calibration

• US 8,355,601 B2
• Filed: 01/15/2010
• Issued: 01/15/2013
• Est. Priority Date: 01/15/2010
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Ding, Yuanyuan, Xiao, Jing
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US12/688,417
• Publication Number:
  US 20110176007A1
• Time in Patent Office: 1,096 Days
• Field of Search: 382/293, 382/294, 382/199, 382/154, 382/295, 382/296, 382/297, 382/298, 348/187, 348/189, 348/580, 353/94, 353/70, 353/69, 345/619, 345/689
• US Class Current: 382/294
• CPC Class Codes: G06T 2207/20041 Distance transform
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 9/3182 Colour adjustment, e.g. white balance, shading or gamut white balance per se H04N9/73; control of amplitude of colour signals H04N9/68; colour control circuits for displays in general G09G5/02, G09G3/2003, G09G3/3607
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

Aspects of the present invention include systems and methods for recalibrating projector-camera systems. In embodiments, systems and methods are able to recalibrate automatically the projector with arbitrary intrinsic and pose, as well as render for arbitrarily desired viewing point. In contrast to previous methods, the methods disclosed herein use the observing camera and the projector to form a stereo pair. Structured light is used to perform pixel-level fine reconstruction of the display surface. In embodiments, the geometric warping is implemented as a direct texture mapping problem. As a result, re-calibration of the projector movement is performed by simply computing the new projection matrix and setting it as a camera matrix. For re-calibrating the new view point, the texture mapping is modified according to the new camera matrix.

Claims

1. At least one non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to execute a computer-implemented method for calibrating a projector system comprising a camera and a projector, each being directed toward a display surface, the computer-implemented method comprising:

obtaining a correspondence that maps a set of projector pixels to a set of the camera pixels via a set of display surface points; and

responsive to the projector being at a different pose;

projecting a set of feature points onto the display surface using the projector, the set of feature points having known projector pixel locations;

capturing at least a portion of the set of feature points using the camera;

detecting a set of detected feature points from the set of feature points;

correlating projector pixels to camera pixels using at least some of the set of detected feature points;

mapping the correlated projector pixels to display surface points from the set of display surface points using the correspondence; and

using the mapped correspondence to obtain a new projection matrix for the projector at the difference pose.

2. The at least one non-transitory computer-readable medium of claim 1 wherein the step of obtaining a correspondence that maps a set of projector pixels to a set of the camera pixels via a set of display surface points comprises:

obtaining a projector projection matrix for the projector and a camera projection matrix for the camera;

using at least an approximation of an inverse light transport matrix to obtain correlation between projector pixels and camera pixels; and

using the correlation between projector pixels and camera pixels, the projector projection matrix, and the camera projection matrix to obtain the set of display surface points and the correspondence.

3. The at least one non-transitory computer-readable medium of claim 2 wherein the step of using the correlation between projector pixels and camera pixels, the projector projection matrix, and the camera projection matrix to obtain the set of display surface points and the correspondence comprises:

back-projecting from a projector image plane and from a camera image plane to triangulate a location for at least some of the display surface points in the set of display surface points.

4. The at least one non-transitory computer-readable medium of claim 2 further comprising the step of:

refining the set of display surface points.

5. The at least one non-transitory computer-readable medium of claim 4 wherein the step of refining the set of display surface points comprises the step of:

identifying a set of outlier points from the set of display surface points;

excluding the set of outlier points from the set of display surface points; and

generating a set of replacement points to replace the set of outlier points that were excluded.

6. The at least one non-transitory computer-readable medium of claim 5 wherein the step of generating a set of replacement points to replace the set of outlier points that were excluded comprises:

for each excluded point from the set of outlier points, using a set of valid display surface points to interpolate a value for the excluded point.

7. The at least one non-transitory computer-readable medium of claim 6 wherein the step of using a set of valid display surface points to interpolate a value for the excluded point comprises:

computing a distance map from at least some of the set of outlier points to a set of non-excluded display surface points; and

for an excluded point from the set of outlier points;

using a local maximum of the distance map to define a local window; and

computing a Delaunay triangulation using at least some of the set of non-excluded display surface points that are within the local window to interpolate a value for the excluded point.

8. The at least one non-transitory computer-readable medium of claim 1 further comprising:

using the new projection matrix to alter an image projected by the projector.

9. At least one non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to execute a computer-implemented method for calibrating a projector system comprising a camera and a projector, each being directed toward a display surface, the computer-implemented method comprising:

obtaining a correspondence that maps a set of projector pixels to a set of the camera pixels via a set of display surface points; and

responsive to having a view location at a position that is different than a position for the camera;

obtaining a new camera projection matrix for the view location; and

using the correspondence to compute a new light projection matrix to alter an image projected by the projector.

10. The at least one non-transitory computer-readable medium of claim 9 wherein the step of using the correspondence to compute a new light projection matrix to alter an image projected by the projector comprises:

setting the new projection matrix as an input of a texture mapper; and

computing texture coordinates for every vertex in a set of display points from the set of display surface points by projecting a vertex to an image plane of the new view location using the new projection matrix.

11. The at least one non-transitory computer-readable medium of claim 9 wherein the step of using the correspondence to compute a new light projection matrix to alter an image projected by the projector comprises:

using the correspondence between the camera and the display surface to assign a color value to each point in a set of display points;

using a projector projection matrix to project the set of display points to a projector image plane; and

assigning pixel color values based upon the projected set of display points.

12. The at least one non-transitory computer-readable medium of claim 9 wherein the step of obtaining a new camera projection matrix for the view location comprises:

obtaining a translation and a rotation of the view location relative to the camera location; and

computing a new projection matrix using the translation and the rotation of the view location relative to the camera and using the camera'"'"'s intrinsic parameters.

13. The at least one non-transitory computer-readable medium of claim 11 further comprising:

receiving the view location information from a user; and

computing the translation and the rotation of the view location relative to the camera location based upon the view location information.

14. The at least one non-transitory computer-readable medium of claim 13 wherein the step of receiving the view location information from a user comprises:

displaying a display environment comprising a depiction of the display surface and the position of the camera; and

receiving the view location information from the user that indicates the view location in the display environment.

15. The at least one non-transitory computer-readable medium of claim 9 wherein the step of obtaining a new camera projection matrix for the view location comprises:

projecting a set of feature points onto the display surface using the projector, the set of feature points having known projector pixel locations;

capturing at least a portion of the set of feature points using the camera;

detecting a set of detected feature points from the set of feature points;

correlating projector pixels to camera pixels using at least some of the set of detected feature points;

mapping the correlated projector pixels to display surface points from the set of display surface points using the correspondence; and

using the mapped correspondence to obtain the new camera projection matrix at the difference pose.

16. The at least one non-transitory computer-readable medium of claim 9 wherein the step of obtaining a correspondence that maps a set of projector pixels to a set of the camera pixels via a set of display surface points comprises:

obtaining a projector projection matrix for the projector and a camera projection matrix for the camera;

using at least an approximation of an inverse light transport matrix to obtain correlation between projector pixels and camera pixels; and

using the correlation between projector pixels and camera pixels, the projection matrix, and the camera projection matrix to obtain the set of display surface points and the correspondence.

17. The at least one non-transitory computer-readable medium of claim 13 wherein the step of using the correlation between projector pixels and camera pixels, the projection matrix, and the camera projection matrix to obtain the set of display surface points and the correspondence comprises:

back-projecting from a projector image plane and from a camera image plane to triangulate a location for at least some of the display surface points in the set of display surface points.

18. A computer system for recalibrating a projector system comprising a projector and a camera, the system comprising:

one or more processors; and

one or more non-transitory computer readable media in communication with the one or more processors and having stored thereon a set of instructions executable by the one or more processors, the set of instructions comprising;

obtaining a correspondence that maps a set of projector pixels to a set of the camera pixels via a set of display surface points; and

responsive to a change of pose of a view location from which an image projected from the projector is viewed by a viewer, using the correspondence to calculate a new camera projection matrix comprising

obtaining a translation and a rotation of the view location relative to the camera location;

computing a new projection matrix using the translation and the rotation of the view location relative to the camera and using the camera'"'"'s intrinsic parameters; and

using the new camera projection matrix to obtain a new light projection matrix for altering an image projected by the projector.

19. A computer system for recalibrating a projector system comprising a projector and a camera, the system comprising:

one or more processors; and

one or more non-transitory computer readable media in communication with the one or more processors and having stored thereon a set of instructions executable by the one or more processors, the set of instructions comprising;

obtaining a correspondence that maps a set of projector pixels to a set of the camera pixels via a set of display surface points; and

responsive to a change of pose of the projector, using the correspondence to calculate a new projection matrix comprising

projecting a set of feature points onto the display surface using the projector, the set of feature points having known projector pixel locations;

capturing at least a portion of the set of feature points using the camera;

detecting a set of detected feature points from the set of feature points;

correlating projector pixels to camera pixels using at least some of the set of detected feature points;

mapping the correlated projector pixels to display surface points from the set of display surface points using the correspondence; and

using the mapped correspondence to obtain the new projection matrix for the projector at the difference pose.