Oblique image stitching

• US 7,778,491 B2
• Filed: 04/10/2006
• Issued: 08/17/2010
• Est. Priority Date: 04/10/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Uyttendaele, Matthew, Steedly, Drew, Cohen, Michael, Szeliski, Richard
• Primary Examiner(s): Bali; Vikkram
• Assistant Examiner(s): ENTEZARI, MICHELLE M
• Application Number:
  US11/279,241
• Publication Number:
  US 20070237420A1
• Time in Patent Office: 1,590 Days
• Field of Search: 382/294, 382/285
• US Class Current: 382/294
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image

Abstract

An “Oblique Image Stitcher” provides a technique for constructing a photorealistic oblique view from a set of input images representing a series of partially overlapping views of a scene. The Oblique Image Stitcher first projects each input image onto a geometric proxy of the scene and renders the images from a desired viewpoint. Once the images have been projected onto the geometric proxy, the rendered images are evaluated to identify optimum seams along which the various images are to be blended. Once the optimum seams are selected, the images are remapped relative to those seams by leaving the mapping unchanged at the seams and interpolating a smooth mapping between the seams. The remapped images are then composited to construct the final mosaiced oblique view of the scene. The result is a mosaic image constructed by warping the input images in a photorealistic manner which agrees at seams between images.

Claims

1. A system for constructing a seamless image of a scene from plurality of oblique images of the scene, comprising using a computing device for:

receiving a set of partially overlapping oblique images of a scene;

texture mapping each of the images to a geometric proxy of the scene;

rendering the texture mapped images to a desired viewpoint;

evaluating the rendered images to identify optimal seams for joining the rendered images; and

compositing the set of rendered images along the optimal seams to construct a seamless image of the scene.

2. The system of claim 1 further comprising:

smoothing the geometric proxy in regions adjacent to the optimal seams; and

remapping the set of oblique input images to the smoothed geometric proxy prior to compositing those images to construct the seamless image of the scene.

3. The system of claim 1 wherein the geometric proxy is automatically constructed from the set of oblique input images.

4. The system of claim 1 wherein evaluating the texture mapped images to identify optimal seams comprises performing a graph-cut analysis to identify a set of lowest cost seams along which to composite the images.

5. The system of claim 4 wherein the graph-cut analysis is biased to prefer seams where the geometric proxy is approximately smooth.

6. The system of claim 4 wherein the graph-cut analysis is biased to prefer seams where images are in best agreement with the geometric proxy.

7. The system of claim 1 wherein further comprising a user interface for interacting with the seamless view of the scene.

8. A computer readable medium having computer executable instructions for automatically constructing a photorealistic mosaiced image from a series of oblique views of a scene, said computer executable instructions comprising:

mapping a set of oblique images of a scene to a 3D model of the scene;

performing a graph-cut analysis of the mapped images to identify optimal seams for joining the mapped images;

attenuating the 3D model of the scene in areas adjacent to the optimal seams while holding areas of the 3D model corresponding to the seams fixed;

remapping the set of oblique images to the attenuated 3D model of the scene; and

compositing the remapped images along the optimal seams to construct a photorealistic mosaiced image of the scene.

9. The computer-readable medium of claim 8 further comprising computer executable instructions for automatically constructing the 3D model from the set of oblique images.

10. The computer-readable medium of claim 8 further comprising a user interface for interacting with the photorealistic mosaiced image of the scene.

11. The computer-readable medium of claim 10 wherein the user interface allows a user to interact with the photorealistic mosaiced image of the scene stored on a server computer from a remote client computer across a network.

12. The computer-readable medium of claim 11 wherein the photorealistic mosaiced image of the scene stored on the server computer is segmented into a plurality of sub-images relative to a predefined grid pattern, and wherein particular sub-images are provided to the user interface on the remote client computer on as as-needed basis.

13. The computer-readable medium of claim 8 wherein the graph-cut analysis is biased to prefer seams where the geometric proxy is approximately smooth.

14. The computer-readable medium of claim 8 wherein the graph-cut analysis is biased to prefer seams where images are in best agreement with the 3D model.

15. The computer-readable medium of claim 8 wherein the 3D model is generated from stereo depth information of the scene derived from the set of oblique images of the scene.

16. A method for constructing a photorealistic novel perspective view from a set of input images representing a series of oblique views of a scene, comprising using a computing device to perform steps for:

texture mapping each of the oblique input images to a geometric proxy representing the scene;

performing a graph-cut analysis of the texture mapped images to identify optimal seams for joining overlapping texture mapped images;

smoothing the geometric proxy in regions adjacent to the seams while holding the geometric proxy fixed in areas directly corresponding to the seams;

remapping the oblique input images to the smoothed geometric proxy; and

compositing the set of remapped images along the optimal seams to construct a photorealistic novel perspective view of the scene.

17. The method of claim 16 further comprising steps for providing a user interface for interacting with the photorealistic view of the scene across a network, said user interface being provided on a client computer while the photorealistic novel perspective view of the scene is maintained on a server computer and provided to the client computer in response to a request from the client computer via the user interface.

18. The method of claim 17 further comprising steps for segmenting the photorealistic novel perspective view of the scene maintained on the server computer into a plurality of sub-images relative to a predefined grid pattern, and wherein particular sub-images are provided to the user interface on the remote client computer on as as-needed basis to facilitate user browsing of the photorealistic novel perspective view of the scene.

19. The method of claim 16 further comprising steps for generating the geometric proxy from stereo depth information of the scene derived from the set of oblique images of the scene.

20. The method of claim 16 further comprising steps for biasing the graph-cut analysis to prefer seams that will enhance the photorealism of the novel perspective view of the scene.