GENERATING GIGAPIXEL IMAGES

• US 20090022421A1
• Filed: 07/18/2007
• Published: 01/22/2009
• Est. Priority Date: 07/18/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Uyttendaele, Matthew T., Cohen, Michael F.
• Granted Patent: US 7,961,983 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images

Abstract

A gigapixel image is generated from a set of images in raw format depicting different portions of a panoramic scene that has up to a full spherical field of view. Radiometric alignment of the images creates a set of images in radiance format. Geometric alignment of the radiance format images creates a set of true poses for the images in radiance format. A gigapixel image depicting the entire scene is assembled from the set of radiance format images and radiance format true poses for the images. The set of images in raw format is captured using a conventional digital camera, equipped with a telephoto lens, attached to a motorized head. The head is programmed to pan and tilt the camera in prescribed increments to individually capture the images at a plurality of exposures and with a prescribed overlap between images depicting adjacent portions of the scene.

Claims

1. A computer-implemented process for generating a gigapixel image, comprising using a computing device to perform the following process actions:

inputting a set of images, wherein the set of images comprises a plurality of captured images in raw pixel data format, each of which depicts a portion of an overall panoramic scene with an amount of overlap between images that depict adjacent portions of the scene;

performing radiometric alignment of each captured image, resulting in the creation of a set of captured images in radiance pixel data format;

performing geometric alignment of each captured image in its radiance pixel data format, resulting in the creation of a set of true poses for the captured images in radiance pixel data format; and

assembling a seamless gigapixel image from the set of captured images in radiance pixel data format and the set of true poses for the captured images in radiance pixel data format.

2. The process of claim 1, wherein the captured images are individually captured at a plurality of exposures.

3. The process of claim 2, wherein the set of images further comprises exposure time information for each captured image in the set, the raw pixel data format comprises linear data values, and the process action of performing radiometric alignment of each captured image comprises the actions of:

de-mosaicing the raw pixel data for each captured image, resulting in the conversion from raw pixel data to color pixel data;

removing lens vignette artifacts from each captured image by multiplying the pixels for each image with a vignette adjustment map;

computing white balance for the panoramic scene; and

converting the pixel data for each captured image from linear data values to radiance data values by dividing the pixel data by the exposure time for the image.

4. The process of claim 3, wherein the process action of de-mosaicing the raw pixel data for each captured image comprises employing a high-quality gradient-corrected linear interpolation technique.

5. The process of claim 3, wherein,

the vignette adjustment map is generated for a particular lens and aperture setting of a camera system used to capture the set of images by using said camera system to capture an image of a very uniform light field, and

the exposure time for the image is defined as a particular shutter speed of a camera system used to capture that image.

6. The process of claim 3, wherein the process action of computing white balance for the panoramic scene comprises the actions of:

selecting a single gray value neutral point for the scene in one of the captured images; and

transforming the gray value neutral point to an enhanced color space neutral point, wherein the gray value neutral point comprises a single pixel in the scene that is void of color but not black.

7. The process of claim 6, wherein the process action of transforming the gray value neutral point to an enhanced color space neutral point comprises transforming the gray value neutral point to an scRGB neutral point using a Bradford chromatic adaptation technique.

8. The process of claim 2, wherein the process action of performing geometric alignment of each captured image in its radiance pixel data format comprises the actions of:

analyzing each captured image for the existence of multi-scale oriented patches (MOPS) features;

whenever it is determined a particular captured image does not contain any MOPS features, considering the pose in said image to be a true pose; and

whenever it is determined a particular captured image does contain particular MOPS features,

extracting the particular MOPS features from said image,

removing radial distortion from the extracted MOPS features by mapping the extracted features through a pre-characterization of radial distortion associated with a particular lens of a camera system used to capture the set of images,

searching a small area surrounding said image for matches to the extracted MOPS features, resulting in the identification of putative matches to the extracted MOPS features, wherein said small area comprises a small segment of each of the up to eight other captured images that overlap said image,

eliminating outliers from the putative matches, resulting in the identification of valid matches to the extracted MOPS feature, and

generating a true pose for said image from the valid matches to the extracted MOPS features.

9. The process of claim 8, wherein,

the process action of eliminating outliers from the identified putative matches to the extracted MOPS features comprises an action of employing a random sample consensus technique, and

the process action of generating a true pose for said image from the valid matches to the extracted MOPS features comprises an action of employing a bundle adjustment technique which is designed to accommodate a large number of images.

10. The process of claim 2, wherein the process action of assembling a seamless gigapixel image comprises the actions of:

down-sampling each captured image in radiance pixel data format;

using the down-sampled images and the true poses for the captured images to create a low resolution proxy composite of the panoramic scene;

using the proxy composite to adjust the exposure of each captured image to improve the radiometric alignment of the captured images, resulting in an adjusted exposure parameter for each captured image;

using the proxy composite to select seams between the captured images and compute, for each pixel in each captured image which overlaps pixels in an adjacent image(s), a low resolution labeling of which image'"'"'s pixel to use in creating the composite gigapixel image;

generating a full resolution labeling for each captured image by expanding the low resolution labeling associated therewith;

using the set of captured images in radiance pixel data format, the set of true poses for the captured images in radiance pixel data format, the adjusted exposure parameter for each captured image, the full resolution labeling for each captured image, a prescribed output mapping for a desired field of view perspective, and a prescribed tile size to warp the captured images into a tiled, high dynamic range (HDR), full resolution, composite gigapixel image that depicts the entire panoramic scene; and

storing each tile of the HDR, full resolution, composite gigapixel image in a prescribed file format.

11. The process of claim 10, wherein the process action of using the proxy composite to adjust the exposure of each captured image comprises an action of using a per color channel gain computation technique.

12. The process of claim 10, wherein the process action of using the proxy composite to select seams between the captured images and compute a low resolution labeling of which image'"'"'s pixel to use in creating the composite gigapixel image comprises an action of using a graph-cut seam selection technique which is iteratively applied to each image.

13. The process of claim 10, wherein the process action of generating a full resolution labeling for each captured image by expanding the low resolution labeling associated therewith comprises a process action of using a joint bilateral up-sampling technique to perform the expansion.

14. The process of claim 10, wherein the desired field of view defines one of an perspective projection, a cylindrical projection, or a spherical projection.

15. The process of claim 10, wherein,

the process action of warping of the captured images into a tiled, HDR, full resolution, composite gigapixel image is performed in a single action so that the captured images are re-sampled only once, and

the prescribed file format comprises 128-bit floating point data in HD Photo file format using the lossless compression feature of the HD Photo format.

16. The process of claim 10, wherein whenever a tone-mapped gigapixel image is desired, the process action of assembling a seamless gigapixel image further comprises the actions of:

using the proxy composite to compute a low resolution tone map;

expanding the low resolution tone map to a full resolution tone map;

applying the full resolution tone map to the tiled, HDR, full resolution, composite gigapixel image resulting in the creation of a tile, tone-mapped, full resolution, composite gigapixel image; and

storing each tile of the tone-mapped, full resolution, composite gigapixel image in a prescribed file format.

17. The process of claim 16, wherein the process action of expanding the low resolution tone map to a full resolution tone map comprises an action of employing a joint bilateral up-sampling technique.

18. The process of claim 16, wherein,

the process action of using the proxy composite to compute a low resolution tone map comprises an action of using an interactive local adjustment of tonal values technique, and

the prescribed file format comprises 24-bit RGB data in HD Photo file format.

19. A system for generating a gigapixel image, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules to,

input a plurality of captured images in raw pixel data format, wherein,

each captured image depicts a different portion of a panoramic scene,

captured images that depict adjacent portions of the panoramic scene overlap each other by a common prescribed amount, and

the raw pixel data format comprises linear data values,

de-mosaic the captured images,

de-vignette the de-mosaiced images,

compute white balance for the panoramic scene,

convert the pixel data for the de-vignetted images from linear data values to radiance data values,

analyze the converted images for the existence of multi-scale oriented patches (MOPS) features,

generate true poses for the converted images using identified MOPS features,

down-sample the converted images,

use the down-sampled converted images and the true poses for the converted images to create a low resolution proxy composite of the panoramic scene,

use the proxy composite to,

adjust exposure of the converted images, resulting in a set of adjusted exposure parameters for the converted images, and

select seams between the converted images and compute a full resolution labeling for each converted image of which pixels, residing in portions of the image that overlap other images, to use in creating the gigapixel image, and

generate a seamless, tiled, high dynamic range, full resolution, composite gigapixel image that depicts the entire panoramic scene from,

the converted images,

the true poses for the converted images,

the set of adjusted exposure parameters for the converted images,

the full resolution labeling for each converted image,

a prescribed output mapping for a desired field of view perspective, and

a prescribed tile size.

20. A system for capturing a set of images depicting a panoramic scene, comprising:

a programmable motorized head;

a digital camera movably attached to the motorized head;

a long telephoto lens rigidly attached to the camera; and

wherein,

the panoramic scene has a field of view comprising one of an perspective projection, a cylindrical projection, or a spherical projection,

the camera'"'"'s aperture is set to a prescribed value which is based on the lens'"'"' modulation transfer function and a desired depth of field,

the camera'"'"'s shutter speed is set by the camera'"'"'s auto-exposure,

the head is programmed to control the camera'"'"'s position, wherein,

the head pans the camera about a vertical axis in prescribed increments and simultaneously tilts the camera about a horizontal axis in prescribed increments in order to individually capture the set of images,

the range of both panning and tilting is up to 360 degrees,

each image depicts a different portion of the panoramic scene,

the images are captured in vertical scan-line order at a plurality of exposures, and

particular images that depict adjacent portions of the panoramic scene overlap each other by a common amount which is prescribed based on the pan and tilt positioning tolerances of the head.