IMAGE STITCHING USING PARTIALLY OVERLAPPING VIEWS OF A SCENE

• US 20100238164A1
• Filed: 06/06/2010
• Published: 09/23/2010
• Est. Priority Date: 04/10/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Uyttendaele, Matthew, Steedly, Drew, Cohen, Michael, Szeliski, Richard
• Granted Patent: US 7,889,948 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image

Abstract

An “Oblique Image Stitcher” provides a technique for constructing a photorealistic oblique view from a set of input images representing a series of partially overlapping views of a scene. The Oblique Image Stitcher first projects each input image onto a geometric proxy of the scene and renders the images from a desired viewpoint. Once the images have been projected onto the geometric proxy, the rendered images are evaluated to identify optimum seams along which the various images are to be blended. Once the optimum seams are selected, the images are remapped relative to those seams by leaving the mapping unchanged at the seams and interpolating a smooth mapping between the seams. The remapped images are then composited to construct the final mosaiced oblique view of the scene. The result is a mosaic image constructed by warping the input images in a photorealistic manner which agrees at seams between images.

Claims

1. A system for constructing a mosaic image of a scene from a plurality of images of the scene, comprising using a computing device for:

receiving a plurality of images of a scene captured from a plurality of viewpoints;

texture mapping a set of the images to a geometric proxy of the scene;

rendering the texture mapped images to a desired view;

identifying one or more seams to join the rendered images; and

constructing a mosaic image by joining the rendered images along the identified seams.

2. The system of claim 1 further comprising:

smoothing the geometric proxy in regions adjacent to the identified seams;

remapping the set of the images to the smoothed geometric proxy prior to constructing the mosaic image;

constructing the mosaic image by joining the remapped images along the identified seams.

3. The system of claim 1 wherein the geometric proxy is automatically constructed from the plurality of images of the scene.

4. The system of claim 1 wherein identifying the seams comprises performing a graph-cut analysis to identify a set of lowest cost seams along which to join the rendered images.

5. The system of claim 4 wherein the graph-cut analysis is biased to prefer seams where the geometric proxy is approximately smooth.

6. The system of claim 4 wherein the graph-cut analysis is biased to prefer seams where images are in best agreement with the geometric proxy

7. The system of claim 1 further comprising a user interface for interacting with the seamless view of the scene.

8. A computer readable storage device having computer executable instructions stored thereon for automatically constructing a photorealistic mosaiced image from a series of partially overlapping images of a scene, said computer executable instructions comprising:

receiving a plurality of images of a scene captured from a plurality of viewpoints;

texture mapping a set of the images to a 3D model of the scene;

identifying seams for joining the texture mapped images;

attenuating the 3D model of the scene in areas adjacent to the identified seams while holding areas of the 3D model corresponding to those seams fixed;

remapping the set of the images to the attenuated 3D model of the scene; and

constructing a photorealistic mosaiced image by compositing the remapped images along the identified seams.

9. The computer-readable medium of claim 8 further comprising computer executable instructions for automatically constructing the 3D model from the plurality of images.

10. The computer-readable medium of claim 8 further comprising a user interface for interacting with the photorealistic mosaiced image of the scene.

11. The computer-readable medium of claim 10 wherein the user interface allows a user to interact with the photorealistic mosaiced image of the scene stored on a server computer from a remote client computer across a network.

12. The computer-readable medium of claim 11 wherein the photorealistic mosaiced image of the scene stored on the server computer is segmented into a plurality of sub-images relative to a predefined grid pattern, and wherein particular sub-images surrounding a currently displayed region of the photorealistic mosaiced image are downloaded to the user'"'"'s computer in the background to accelerate local loading of the sub-images during user panning of the photorealistic mosaiced image.

13. The computer-readable medium of claim 8 wherein identification of seams is performed using a graph-cut analysis that is biased to prefer seams where the 3D model is approximately smooth.

14. The computer-readable medium of claim 8 wherein identification of seams is performed using a graph-cut analysis that is biased to prefer seams where images are in best agreement with the 3D model.

15. The computer-readable medium of claim 8 wherein the 3D model is generated from stereo depth information of the scene derived from the plurality of images of the scene.

16. A method for constructing a photorealistic view from a set of input images of a scene, comprising steps for:

receiving a plurality of images of a scene, wherein one or more of the images partially overlap others of the plurality of images;

texture mapping a set of the images to a 3D model of the scene;

rendering the texture mapped images to a specified view;

identifying one or more seams to join the rendered images; and

constructing a photorealistic view of the scene by compositing the rendered images along the identified seams.

17. The method of claim 16 further comprising:

smoothing the 3D model in regions adjacent to the identified seams; and

remapping the set of the images to the smoothed 3D model prior to constructing the photorealistic view of the scene.

18. The method of claim 16 wherein the 3D model is automatically constructed from the plurality of images.

19. The method of claim 16 wherein identifying the seams comprises performing a graph-cut analysis to identify a set of lowest cost seams along which to join the rendered images.

20. The method of claim 19 wherein the graph-cut analysis is biased to prefer seams where the 3D model is approximately smooth.