Multi-projector mosaic with automatic registration

• US 6,733,138 B2
• Filed: 08/15/2001
• Issued: 05/11/2004
• Est. Priority Date: 08/15/2001
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Raskar, Ramesh
• Primary Examiner(s): Fuller, Rodney
• Assistant Examiner(s): Koval, Melissa J
• Application Number:
  US09/930,425
• Publication Number:
  US 20030052837A1
• Time in Patent Office: 1,000 Days
• Field of Search: 353/94, 353/69, 353/70, 353/121, 353/122, 345/32, 382/154
• US Class Current: 353/94
• CPC Class Codes: H04N 9/12 Picture reproducers H04N9/11 takes precedence
• CPC Class Codes: H04N 9/3147 Multi-projection systems displays in general H04N9/12; video walls G06F3/1446, G09G2300/026
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

A method forms a mosaic image on a display surface with a multiple projectors. For each projector in turn, a registration image is projected onto the display surface so that a union of the projected registration images forms a polygon. With a camera, for each registration image in turn, a corresponding input image is acquired. A display area on the display surface enclosed by the polygon is then identified, and a single projective matrix between the display area and each input image is determined for each projector. A source image for each projector is warped according to the corresponding homography of the projector. The pixels of the warped source image are weighted according to the single projective matrix, and then the warped and weighted source images are concurrently projected directly onto the display surface to form the mosaic image.

Claims

1. A method for forming a mosaic image on a display surface with a plurality of projectors, comprising:

projecting, for each projector in turn, a single registration image onto the display surface so that a union of the projected registration images forms a polygon;

acquiring with a camera, for each single registration image in turn, a corresponding input image;

identifying a display area on the display surface enclosed by the polygon;

determining, for each projector, a single projective matrix between the display area and each input image, wherein the single projective matrix is a homography;

warping, for each projector, a source image according to the corresponding single projective matrix;

weighting pixels of the warped source image according to the single projective matrix; and

concurrently projecting the warped and weighted source images directly onto the display surface to form the mosaic image.

2. The method of claim 1 wherein the display surface is oblique to an optical axis of at least one of the projectors.

3. The method of claim 1 wherein an optical axis of one projector is oblique to an optical axis of at least one other projector.

4. The method of claim 1 wherein an optical axis of each projector is oblique an optical axis of every other projector.

5. The method of claim 1 wherein the registration image includes a checkerboard pattern.

6. The method of claim 1 wherein the polygon encloses the display area and the display area is a largest possible rectangle that is enclosed by the polygon.

7. The method of claim 1 further comprising:

defining, for each projector, a first homography between the camera and the projector;

defining, for each projector, a second homography between the display area and the camera; and

combining the first and second homographies to form the single projective matrix.

8. The method of claim 7 wherein the first homography H_i is a maximum likelihood estimation problem, and further comprising:

minimizing a cost function $\stackrel{n}{\sum <br><br>}<br><br><br><br>u-\mathrm{Hx}<br><br>,$

where n indicates a number of corresponding features between the input image x, and the registration image u.

9. The method of claim 1 further comprising:

assigning a zero weight to a particular pixel in each warped image if the particular pixel is outside the display area;

assigning a one weight to the particular pixel if the particular pixel is an only pixel illuminating the display area; and

otherwise

assigning a weight W in a range 0<

W<

1 to the particular pixel, where W is proportional to a distance to a nearest pixel having a zero weight.

10. A system for forming a mosaic image on a display surface, comprising:

a plurality of projectors, each projector arranged to project a single registration image onto the display surface so that a union of the projected registration images forms a polygon;

a camera arranged to acquiring, for each single registration image in turn, a corresponding input image;

means for identifying a display area on the display surface enclosed by the polygon;

means for determining, for each projector, a single projective matrix between the display area and each input image, wherein the single projective matrix is a homography;

means for warping, for each projector, a source image according to the corresponding single projective matrix;

means for weighting pixels of each warped source image according to the single projective matrix so that projected warped and weighted source images form the mosaic image on the display surface.