PROCESSING MULTI-VIEW DIGITAL IMAGES

• US 20100271511A1
• Filed: 04/19/2010
• Published: 10/28/2010
• Est. Priority Date: 04/24/2009
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Ma, Zhonghua, Lakus-Becker, Axel, Leung, Ka-Ming
• Granted Patent: US 8,509,558 B2
• Time in Patent Office: Days
• US Class Current: 348/239
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

Methods, apparatuses (100, 400, 1000), and computer program products for generating an enhanced digital image (490, 495, 1022) comprising a plurality of pixels are disclosed. Using a first digital image (420, 1020) captured from a first camera (124) and parity bits (410, 415, 1010) generated from a second digital image captured by a second camera (122, 126), a third digital image (445, 447, 1045) is constructed. The second camera (122, 126) captures the second image at a resolution different to the resolution of the first camera (124) capturing the first image (420, 1020). A disparity map (455, 457, 1055) between the first image (420, 1020) and the third image (445, 447, 1045) is determined (450, 452, 1050). One of the first image (420, 1020) and the third image (445, 447, 1045) is enhanced (470, 472, 1070) dependent upon the determined disparity map (455, 457, 1055) to generate the enhanced digital image (490, 495, 1022).

Claims

1. A method of generating an enhanced digital image comprising a plurality of pixels, said method comprising the steps of:

using a first digital image captured from a first camera and parity bits generated from a second digital image captured by a second camera to construct a third digital image, the second camera capturing said second image at a resolution different to the resolution of said first camera capturing said first image;

determining a disparity map dependent upon said first image and said third image; and

enhancing one of said first image and said third image dependent upon said determined disparity map to generate said enhanced digital image.

2. The method as claimed in claim 1, further comprising the step of storing one or more of said first digital image, said third digital image, and said enhanced digital image.

3. The method as claimed in claim 1, further comprising the steps of:

obtaining said first image from said first camera; and

obtaining said parity bits generated from said second image captured by said second camera.

4. The method as claimed in claim 1, wherein said first image is captured at a higher resolution than said second image.

5. The method as claimed in claim 1, wherein said first and second images are captured by said first and second cameras, respectively, having overlapping views of a scene.

6. The method as claimed in claim 1, further comprising the step of downsampling said first digital image to a resolution matching the resolution of said second digital image.

7. The method as claimed in claim 6, further comprising the step of Wyner-Ziv decoding said parity bits generated from said second digital image using said downsampled first image to provide said third image, said parity bits output by a Wyner-Ziv encoder coupled to said second camera.

8. The method as claimed in claim 1, wherein said disparity map is determined using a downsampled version of said first image and said third image to provide a disparity map between said downsampled version of said first image and said third image.

9. The method as claimed in claim 1, further comprising the step of interpolating said disparity map to provide a higher-resolution version of said disparity map for use in said enhancing step.

10. The method as claimed in claim 9, wherein said enhancing step comprises disparity compensating said first image using said interpolated disparity map to generate said enhanced digital image, said enhanced digital image being a high resolution version of said second image.

11. The method as claimed in claim 10, wherein said disparity compensating comprises, for each pixel location in said third image:

extracting a corresponding disparity value from said interpolated disparity map;

locating a reference pixel in said first image; and

replacing a pixel value of said pixel location in said third image with said pixel value of said located reference pixel.

12. The method as claimed in claim 1, further comprising the step of partitioning said first image into a plurality of image segments dependent upon said disparity map and said third image, each segment comprising an object having properties.

13. The method as claimed in claim 12, wherein said properties of said objects in said segments are used to enhance said first image in said enhancing step.

14. The method as claimed in claim 13, further comprising the step of identifying said objects based on properties of said objects from a database of pre-defined object primitives, said predefined object primitives used in said enhancing step to enhance said first image.

15. The method as claimed in claim 1, further comprising the steps of:

using said first digital image captured from said first camera and parity bits generated from a fourth digital image captured by a third camera, constructing a fifth digital image, the third camera capturing said fourth image at a resolution different to the resolution of said first camera capturing said first image;

determining a disparity map dependent upon said first image and said fifth image; and

enhancing one of said first image and said fifth image dependent upon said determined disparity map to generate said enhanced digital image.

16. An apparatus for generating an enhanced digital image comprising a plurality of pixels, said apparatus comprising:

a memory for storing data and a computer program for a processor; and

a processor coupled to said memory for executing said computer program stored in said memory to provide said enhanced digital image, said computer program comprising;

computer program code means for using a first digital image captured from a first camera and parity bits generated from a second digital image captured by a second camera to construct a third digital image, the second camera capturing said second image at a resolution different to the resolution of said first camera capturing said first image;

computer program code means for determining a disparity map dependent upon said first image and said third image; and

computer program code means for enhancing one of said first image and said third image dependent upon said determined disparity map to generate said enhanced digital image.

17. The apparatus as claimed in claim 16, further comprising:

a first camera for capturing said first image;

a second camera for capturing said second image;

an encoder coupled to said second camera for generating said parity bits from said second image; and

a communication interface coupled to said memory and said processor for providing said first image and said parity bits from said second image to said memory and said processor.

18. An image processing apparatus for generating an enhanced digital image comprising a plurality of pixels, said apparatus comprising:

a constructing unit configured to construct using a first image captured from a first camera and parity bits generated from a second camera a third image, said second camera capturing said second image at a resolution different to the resolution of said first camera capturing said first image;

a determining unit configured to determining a disparity map dependent upon said first image and said third image; and

an enhancing unit configured to enhance one of said first image and said third image dependent upon said determined disparity map to generate said enhanced digital image.

19. The apparatus according to claim 18, further comprising an obtaining unit configured to obtain said first image from said first camera and to obtain parity bits generated from said second image captured by said second camera.

20. A computer program product comprising a computer readable medium having recorded thereon a computer program for generating an enhanced digital image comprising a plurality of pixels, said computer program:

computer program code means for using a first digital image captured from a first camera and parity bits generated from a second digital image captured by a second camera to construct a third digital image, the second camera capturing said second image at a resolution different to the resolution of said first camera capturing said first image;

computer program code means for determining a disparity map dependent upon said first image and said third image; and

computer program code means for enhancing one of said first image and said third image dependent upon said determined disparity map to generate said enhanced digital image.