Processing multi-view digital images

• US 8,509,558 B2
• Filed: 04/19/2010
• Issued: 08/13/2013
• Est. Priority Date: 04/24/2009
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Ma, Zhonghua, Lakus-Becker, Axel, Leung, Ka-Ming
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US12/762,617
• Publication Number:
  US 20100271511A1
• Time in Patent Office: 1,212 Days
• Field of Search: 382/299, 382/254, 382/154, 382/286, 348/239, 348135-149
• US Class Current: 382/254
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

Methods, apparatuses (100, 400, 1000), and computer program products for generating an enhanced digital image (490, 495, 1022) comprising a plurality of pixels are disclosed. Using a first digital image (420, 1020) captured from a first camera (124) and parity bits (410, 415, 1010) generated from a second digital image captured by a second camera (122, 126), a third digital image (445, 447, 1045) is constructed. The second camera (122, 126) captures the second image at a resolution different to the resolution of the first camera (124) capturing the first image (420, 1020). A disparity map (455, 457, 1055) between the first image (420, 1020) and the third image (445, 447, 1045) is determined (450, 452, 1050). One of the first image (420, 1020) and the third image (445, 447, 1045) is enhanced (470, 472, 1070) dependent upon the determined disparity map (455, 457, 1055) to generate the enhanced digital image (490, 495, 1022).

Claims

1. A method of generating an enhanced digital image comprising a plurality of pixels, said method comprising the steps of:

using a first digital image captured by a first camera and parity bits generated from a second digital image captured by a second camera to construct a third digital image, the third digital image having a viewpoint matching the second camera, the second camera capturing the second digital image at a resolution lower than the resolution of the first camera capturing the first digital image;

determining a disparity map dependent upon the first digital image and the third digital image; and

enhancing the third digital image dependent upon the determined disparity map and the first digital image to generate the enhanced digital image, the enhanced digital image having a viewpoint corresponding to a viewpoint of the third digital image and having a resolution greater than the second digital image.

2. The method as claimed in claim 1, further comprising the step of storing one or more of the first digital image, the third digital image, and the enhanced digital image.

3. The method as claimed in claim 1, further comprising the steps of:

obtaining the first digital image from the first camera; and

obtaining the parity bits generated from the second digital image captured by the second camera.

4. The method as claimed in claim 1, wherein the first digital image is captured at a higher resolution than the second digital image.

5. The method as claimed in claim 1, wherein the first and second digital images are captured by the first and second cameras, respectively, having overlapping views of a scene.

6. The method as claimed in claim 1, further comprising the step of downsampling the first digital image to a resolution matching the resolution of the second digital image.

7. The method as claimed in claim 6, further comprising the step of Wyner-Ziv decoding the parity bits generated from the second digital image using the downsampled first digital image to provide the third digital image, the parity bits being output by a Wyner-Ziv encoder coupled to the second camera.

8. The method as claimed in claim 1, wherein the disparity map is determined using a downsampled version of the first digital image and the third digital image to provide a disparity map between the downsampled version of the first digital image and the third digital image.

9. The method as claimed in claim 1, further comprising the step of interpolating the disparity map to provide a higher-resolution version of the disparity map for use in said enhancing step.

10. The method as claimed in claim 9, wherein said enhancing step comprises disparity compensating the first digital image using the interpolated disparity map to generate the enhanced digital image, the enhanced digital image being a high resolution version of the second digital image.

11. The method as claimed in claim 10, wherein said disparity compensating comprises, for each pixel location in the third digital image:

extracting a corresponding disparity value from the interpolated disparity map;

locating a reference pixel in the first digital image; and

replacing a pixel value of the pixel location in the third digital image with the pixel value of the located reference pixel.

12. The method as claimed in claim 1, further comprising the step of partitioning the first digital image into a plurality of image segments dependent upon the disparity map and the third digital image, each segment comprising an object having properties.

13. The method as claimed in claim 12, wherein the properties of the objects in the segments are used to enhance the first digital image in said enhancing step.

14. The method as claimed in claim 13, further comprising the step of identifying the objects based on properties of the objects from a database of pre-defined object primitives, the predefined object primitives being used in said enhancing step to enhance the first digital image.

15. The method as claimed in claim 1, further comprising the steps of:

using the first digital image captured by the first camera and parity bits generated from a fourth digital image captured by a third camera to construct a fifth digital image, the fifth digital image having a viewpoint matching the third camera, the third camera capturing the fourth digital image at a resolution lower than the resolution of the first camera capturing the first digital image;

determining a disparity map dependent upon the first digital image and the fifth digital image; and

enhancing the fifth digital image dependent upon the determined disparity map and the first digital image to generate the enhanced digital image, the enhanced digital image having a viewpoint corresponding to a viewpoint of the fifth digital image and having a resolution greater than the fourth digital image.

16. An apparatus for generating an enhanced digital image comprising a plurality of pixels, said apparatus comprising:

a memory for storing data and a computer program for a processor; and

a processor coupled to said memory for executing the computer program stored in said memory to provide the enhanced digital image, the computer program comprising;

(a) computer program code means for using a first digital image captured by a first camera and parity bits generated from a second digital image captured by a second camera to construct a third digital image, the third digital image having a viewpoint matching the second camera, the second camera capturing the second digital image at a resolution lower than the resolution of the first camera capturing the first digital image;

(b) computer program code means for determining a disparity map dependent upon the first digital image and the third digital image; and

(c) computer program code means for enhancing the third image dependent upon the determined disparity map and the first digital image to generate the enhanced digital image, the enhanced digital image having a viewpoint corresponding to a viewpoint of the third digital image and having a resolution greater than the second digital image.

17. The apparatus as claimed in claim 16, further comprising:

a first camera for capturing the first digital image;

a second camera for capturing the second digital image;

an encoder coupled to said second camera for generating the parity bits from the second digital image; and

a communication interface coupled to said memory and said processor for providing the first digital image and the parity bits from the second digital image to said memory and said processor.

18. An image processing apparatus for generating an enhanced digital image comprising a plurality of pixels, said apparatus comprising:

a constructing unit configured to construct, using a first image captured by a first camera and parity bits generated from a second camera, a third digital image, the third digital image having a viewpoint matching the second camera, the second camera capturing the second digital image at a resolution lower than the resolution of the first camera capturing the first digital image;

a determining unit configured to determine a disparity map dependent upon the first digital image and the third digital image; and

an enhancing unit configured to enhance the third digital image dependent upon the determined disparity map and the first digital image to generate the enhanced digital image, the enhanced digital image having a viewpoint corresponding to a viewpoint of the third digital image and having a resolution greater than the second digital image.

19. The apparatus according to claim 18, further comprising an obtaining unit configured to obtain the first digital image from the first camera and to obtain parity bits generated from the second digital image captured by the second camera.

20. A computer program product comprising a non-transitory computer-readable storage medium having recorded thereon a computer program for generating an enhanced digital image comprising a plurality of pixels, the computer program comprising:

computer program code means for using a first digital image captured by a first camera and parity bits generated from a second digital image captured by a second camera to construct a third digital image, the third digital image having a viewpoint matching the second camera, the second camera capturing the second digital image at a resolution lower than the resolution of said the camera capturing the first digital image;

computer program code means for determining a disparity map dependent upon the first digital image and the third digital image; and

computer program code means for enhancing the third image dependent upon the determined disparity map and the first digital image to generate the enhanced digital image, the enhanced digital image having a viewpoint corresponding to a viewpoint of the third digital image and having a resolution greater than the second digital image.