Real-time preview for panoramic images

• US 7,424,218 B2
• Filed: 07/28/2005
• Issued: 09/09/2008
• Est. Priority Date: 07/28/2005
• Status: Active Grant

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Uyttendaele, Matthew, Steedly, Drew, Szeliski, Richard, Baudisch, Patrick, Tan, Desney, Rudolph, Eric, Pal, Chris
• Primary Examiner(s): Perkey; W. B.
• Assistant Examiner(s): Suthar; Rishi S
• Application Number:
  US11/193,091
• Publication Number:
  US 20070025723A1
• Time in Patent Office: 1,139 Days
• Field of Search: 396/322, 396/373, 396/374, 396/439, 352/70, 348/36, 348/218.1, 34833301-33305, 348/333.11, 348/333.12
• US Class Current: 396/322
• CPC Class Codes: G03B 13/02 Viewfinders
• CPC Class Codes: G03B 17/18 Signals indicating condition of a camera member or suitability of light indicating depth of field G03B13/30
• CPC Class Codes: H04N 23/634 Warning indications
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Abstract

A “Panoramic Viewfinder” provides an intuitive interactive viewfinder display which operates on a digital camera display screen. This interactive viewfinder provides real-time assistance in capturing images for constructing panoramic image mosaics. The Panoramic Viewfinder “brushes” a panorama from images captured in any order, while providing visual feedback to the user for ensuring that desired scene elements will appear in the final panorama. This visual feedback presents real-time stitched previews of the panorama while capturing images. In one embodiment, the viewfinder display of the Panoramic Viewfinder includes a “mosaic preview” which presents a stitched mosaic preview of the captured images; a live display window representing a “current content” of the camera viewfinder, which is mapped to a matching location within the mosaic preview; and an optional panoramic “cropping frame” overlaid onto the mosaic preview which illustrates a section of the mosaic which will survive a rectangular cropping of the mosaic.

Claims

1. A camera for automatically generating mosaic images from a set of images of a scene, comprising:

a device for performing real-time generation of a mosaic image from images captured by the camera;

a display screen for displaying the mosaic image;

a display window showing a current view through the camera lens of a next image frame to be possibly captured by the camera, said display window being automatically mapped in real-time to a corresponding location on the mosaic image being displayed on the display screen; and

a device for automatically displaying, on the display screen, alerts regarding a loss of tracking between the display window showing the current view through the camera lens and the mosaic image being displayed on the display screen, said alerts including suggestions as to how to reacquire the tracking for automatically mapping the display window to the corresponding location on the mosaic image.

2. The camera of claim 1 further comprising a device for mapping a cropping frame to the mosaic image.

3. The camera of claim 2 wherein the cropping frame represents extents of a largest possible continuous panorama that can be extracted from the mosaic without allowing gaps of image data along any border of the cropping frame.

4. The camera of claim 1 further comprising a device for automatically displaying, on the display device, warnings regarding flaws in the mosaic image.

5. The camera of claim 1 further comprising a device for automatically triggering image capture as function of a computed image overlap as a user pans the camera over a scene.

6. The camera of claim 1 further comprising a device for automatically identifying and deleting redundant image information as each image is added to the mosaic image during real-time generation of the mosaic image.

7. The camera of claim 1 further comprising a device for automatically scaling the mosaic image to fit the extents of the display device as images are added to the mosaic in real-time.

8. The camera of claim 7 further comprising a device for automatically scaling the display window mapped to the mosaic image as a function of the scaling of the mosaic image.

9. A computer readable medium having computer executable instructions for automatically generating real-time mosaic preview images, said computer executable instructions comprising:

receiving a plurality of input images of a scene from an input camera;

constructing a mosaic image in real-time from the input images as each image is received from the input camera and displaying the mosaic image on a display device coupled to the input camera;

displaying a current view of the input camera in real-time as a mapped overlay on the displayed mosaic image;

displaying an automatically sized cropping frame as a mapped overlay on the mosaic image, said cropping frame illustrating an area of the mosaic image that will remain following automatic cropping of the mosaic image; and

wherein the cropping frame represents extents of a largest possible panorama that can be extracted from the mosaic image without allowing gaps of image data along any border of the cropping frame.

10. The computer-readable medium of claim 9 further comprising automatically detecting defects in the mosaic image, and providing automatic alerts via the display device, said alerts being specific to a type of defect that is identified.

11. The computer-readable medium of claim 10 wherein the type of defect that is identified includes any of:

ghosting related defects;

image gap related defects; and

loss of tracking defects relating to any of mapping of the current view of the input camera to the mosaic image and mapping one of the input images to the mosaic image for use in constructing the mosaic image.

12. The computer-readable medium of claim 9 wherein the mosaic image that is constructed is a low-fidelity preview image, and wherein preliminary image stitching information regarding spatial relationship data between the input images used to construct the low-fidelity preview image is stored and used to initialize post-hoc stitching of the input images to construct a final mosaic image.

13. The computer-readable medium of claim 9 wherein the mosaic image is a final mosaic image, and wherein the input images used to construct the final mosaic image are automatically discarded as each image is added to the final mosaic image.

14. The computer-readable medium of claim 9 further comprising automatically triggering image capture from the input camera as function of a computed image overlap as the camera is panned over a scene.

15. A system for generating mosaic images via an interactive viewfinder display comprising:

providing a plurality of input images from a camera device to a computing device in real-time as the images are captured by the camera device, said images not having any predefined spatial relationships and said camera device having an arbitrary position with respect to each image;

constructing a mosaic image in real-time from the input images as each image is received by the computing device from the camera device;

displaying the mosaic image on a display device coupled to the computing device in real-time as the mosaic is constructed in real-time from the input images;

displaying a viewfinder window showing a current view through a camera lens of the camera device, said viewfinder window being automatically mapped in real-time into a corresponding location on the mosaic image being displayed on the display device; and

displaying an automatically sized cropping frame as a mapped overlay on the mosaic image, said cropping frame illustrating an area of the mosaic image that will remain following automatic cropping of the mosaic image.

16. The system of claim 15 wherein the computing device is integral to the camera.

17. The system of claim 15 further comprising automatically identifying and deleting redundant image information from the input images as each image is added to the mosaic image during real-time construction of the mosaic image.

18. The system of claim 15 wherein the mosaic image is a 360-degree mosaic image.

19. The system of claim 15 further comprising automatically detecting defects in the mosaic image, and providing automatic alerts via the display device, said alerts being specific to a type of defect that is identified.