Panorama Imaging Using a Blending Map

• US 20110141300A1
• Filed: 12/11/2009
• Published: 06/16/2011
• Est. Priority Date: 12/11/2009
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Ireland Limited (Adeia Inc.)
• Inventors: Drimbarean, Alexandru, Corcoran, Peter, Barcovschi, Igor, Murray, Larry, Stec, Piotr
• Granted Patent: US 8,294,748 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A technique of generating a panoramic image involves acquiring a set of at least two main image frames, as well as corresponding generating and/or acquiring corresponding relatively low-resolution image frames, of overlapping portions of a panoramic scene. A map or other information is stored relating to the generation of a low-res panorama. A main panorama image is formed by joining main image frames based on the map or other information gained in the low-res process.

Claims

1. A method of generating a panorama image using a portable imaging device, said method comprising:

fixing an exposure level for acquiring the panorama image using the portable imaging device;

panning the imaging device across a panoramic scene,

acquiring a set of at least two image frames of overlapping portions of said panorama image and processing said set, said acquiring comprising using an optic and imaging sensor of the portable imaging device, said processing including;

sorting and retaining a set of image frames including one or more overlapping pairs of image frames,

determining a relative displacement between each of said set of overlapping image frames, including determining the overlapped region for each image pair,

registering or aligning, or both, said set of images based on the relative displacements;

determining an alpha blending map or an optimal stitch line or both for each pair of overlapping image frames;

joining said one or more pairs of image frames to form a panorama image; and

storing, transmitting or displaying said panorama image, or combinations thereof.

2. The method of claim 1 wherein the determining of relative displacement comprises determining relative horizontal displacement between a pair of images of a set of overlapping image frames, and further includes summing pixel values in image columns across each of first and second images to determine a horizontal image profile for each image, and then determining a column difference profile across each of said images, and then determining a relative error function between said pair of images that is determined from said image column difference profiles, and wherein a minimum of said relative error function indicates a relative number of pixel columns of horizontal displacement are between said pair of images.

3. The method of claim 2, wherein the determining of relative displacement comprises determining relative vertical displacement between the pair of images of the set of overlapping image frames, and further comprises summing pixel values in image rows across each of first and second images to determine a vertical image profile for each image, and then determining a row difference profile across each of said images, and then determining a relative error function between said pair of images that is determined from said image row difference profiles, and wherein a minimum of said relative error function indicates a relative number of pixel rows of vertical displacement are between said pair of images.

4. The method of claim 3, further comprising applying a smoothing function to the column or row difference profile, or both, of each image prior to calculating the relative error function between the pair of images.

5. The method of claim 2, further comprising applying a smoothing function to the column difference profile of each image prior to calculating the relative error function between the pair of images.

6. The method of claim 1 where said joining is performed using said alpha blending map or said optimal stitch line, or both.

7. The method of claim 1 where said joining is based on determining both said approximately optimal stitch line and said alpha blending map, and wherein said approximately optimal stitch line comprises an approximately 50% blending ratio between overlapped pixels of the pair of images, and wherein said alpha blending map provides a blending ratio for overlapped pixels from the pair of images in the vicinity of said approximately optimal stitch line.

8. The method of claim 1, further comprising interleaving joining of pairs of image frames with acquiring or generating, or both, of next image frames.

9. The method of claim 1, further comprising cropping said panorama image to a uniform vertical height.

10. A portable camera-enabled device capable of in-camera generation of a panorama image, comprising:

a lens;

an image sensor;

a processor; and

a processor readable medium having code embedded therein for programming the processor to perform a panorama image generation method that comprises;

fixing an exposure level for acquiring the panorama image using the portable device;

panning the imaging device across a panoramic scene,

acquiring a set of at least two image frames of overlapping portions of said panorama image and processing said set, said acquiring comprising using an optic and imaging sensor of the portable imaging device, said processing including;

sorting and retaining a set of image frames including one or more overlapping pairs of image frames,

determining a relative displacement between each of said set of overlapping image frames, including determining the overlapped region for each image pair, registering or aligning, or both, said set of images based on the relative displacements;

determining an alpha blending map or an optimal stitch line or both for each pair of overlapping image frames;

joining said one or more pairs of image frames to form a panorama image; and

storing, transmitting or displaying said panorama image, or combinations thereof.

11. The device of claim 10, wherein the determining of relative displacement comprises determining relative horizontal displacement between a pair of images of a set of overlapping image frames, and further includes summing pixel values in image columns across each of first and second images to determine a horizontal image profile for each image, and then determining a column difference profile across each of said images, and then determining a relative error function between said pair of images that is determined from said image column difference profiles, and wherein a minimum of said relative error function indicates a relative number of pixel columns of horizontal displacement are between said pair of images.

12. The device of claim 11, wherein the determining of relative displacement comprises determining relative vertical displacement between the pair of images of the set of overlapping image frames, and further comprises summing pixel values in image rows across each of first and second images to determine a vertical image profile for each image, and then determining a row difference profile across each of said images, and then determining a relative error function between said pair of images that is determined from said image row difference profiles, and wherein a minimum of said relative error function indicates a relative number of pixel rows of vertical displacement are between said pair of images.

13. The device of claim 12, wherein the method further comprises applying a smoothing function to the column or row difference profile, or both, of each image prior to calculating the relative error function between the pair of images.

14. The device of claim 11, wherein the method further comprises applying a smoothing function to the column difference profile of each image prior to calculating the relative error function between the pair of images.

15. The device of claim 10, wherein said joining is performed using said alpha blending map or said optimal stitch line, or both.

16. The device of claim 10, wherein said joining is based on determining both said approximately optimal stitch line and said alpha blending map, and wherein said approximately optimal stitch line comprises an approximately 50% blending ratio between overlapped pixels of the pair of images, and wherein said alpha blending map provides a blending ratio for overlapped pixels from the pair of images in the vicinity of said approximately optimal stitch line.

17. The device of claim 10, wherein the method further comprises interleaving joining of pairs of image frames with acquiring or generating, or both, of next image frames.

18. The device of claim 10, wherein the method further comprises cropping said panorama image to a uniform vertical height.

19. One or more processor-readable storage media having code embedded therein for programming a processor to perform a method for acquiring a panorama image using a portable imaging device, said method comprising:

fixing an exposure level for acquiring the panorama image using the portable imaging device;

panning the imaging device across a panoramic scene,

acquiring a set of at least two image frames of overlapping portions of said panorama image and processing said set, said acquiring comprising using an optic and imaging sensor of the portable imaging device, said processing including;

sorting and retaining a set of image frames including one or more overlapping pairs of image frames,

determining a relative displacement between each of said set of overlapping image frames, including determining the overlapped region for each image pair,

registering or aligning, or both, said set of images based on the relative displacements;

determining an alpha blending map or an optimal stitch line or both for each pair of overlapping image frames;

joining said one or more pairs of image frames to form a panorama image; and

storing, transmitting or displaying said panorama image, or combinations thereof.

20. The one or more processor-readable storage media of claim 19, wherein the determining of relative displacement comprises determining relative horizontal displacement between a pair of images of a set of overlapping image frames, and further includes summing pixel values in image columns across each of first and second images to determine a horizontal image profile for each image, and then determining a column difference profile across each of said images, and then determining a relative error function between said pair of images that is determined from said image column difference profiles, and wherein a minimum of said relative error function indicates a relative number of pixel columns of horizontal displacement are between said pair of images.

21. The one or more processor-readable storage media of claim 20, wherein the determining of relative displacement comprises determining relative vertical displacement between the pair of images of the set of overlapping image frames, and further comprises summing pixel values in image rows across each of first and second images to determine a vertical image profile for each image, and then determining a row difference profile across each of said images, and then determining a relative error function between said pair of images that is determined from said image row difference profiles, and wherein a minimum of said relative error function indicates a relative number of pixel rows of vertical displacement are between said pair of images.

22. The one or more processor-readable storage media of claim 21, wherein the method further comprises applying a smoothing function to the column or row difference profile, or both, of each image prior to calculating the relative error function between the pair of images.

23. The one or more processor-readable storage media of claim 20, wherein the method further comprises applying a smoothing function to the column difference profile of each image prior to calculating the relative error function between the pair of images.

24. The one or more processor-readable storage media of claim 19, wherein said joining is performed using said alpha blending map or said optimal stitch line, or both.

25. The one or more processor-readable storage media of claim 19, wherein said joining is based on determining both said approximately optimal stitch line and said alpha blending map, and wherein said approximately optimal stitch line comprises an approximately 50% blending ratio between overlapped pixels of the pair of images, and wherein said alpha blending map provides a blending ratio for overlapped pixels from the pair of images in the vicinity of said approximately optimal stitch line.

26. The one or more processor-readable storage media of claim 19, wherein the method further comprises interleaving joining of pairs of image frames with acquiring or generating, or both, of next image frames.

27. The one or more processor-readable storage media of claim 19, wherein the method further comprises cropping said panorama image to a uniform vertical height.