IMAGE PROCESSING APPARATUS AND IMAGE PROCESSING METHOD

• US 20160105660A1
• Filed: 05/23/2014
• Published: 04/14/2016
• Est. Priority Date: 06/05/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: ITO, Katsuhisa, HAYASAKA, Kengo, MORI, Hironori
• Granted Patent: US 10,027,947 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/225 using parallax barriers
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

The present disclosure relates to an image processing apparatus and an image processing method that enable high-precision generation of parallax images of viewpoints with a small amount of calculation.

A correlation generating unit calculates correlation values of captured images of pairs of viewpoints, using captured images of viewpoints. A generating unit generates parallax images of different hierarchical levels of a reference viewpoint based on the correlation values. The parallax images are a parallax image formed with the disparity value of the reference viewpoint, and a parallax image formed with the disparity value of an occlusion region. The reference viewpoint is one of the viewpoints. The present disclosure can be applied to the image processing apparatus of a light field camera, for example.

Claims

1. An image processing apparatus comprising:

a calculating unit configured to calculate correlation values of captured images of a plurality of pairs of viewpoints, using captured images of the viewpoints; and

a generating unit configured to generate parallax images of different hierarchical levels of a reference viewpoint based on the correlation values calculated by the calculating unit, the parallax images being a viewpoint parallax image formed with a disparity value of the reference viewpoint and an occlusion parallax image formed with a disparity value of an occlusion region, the reference viewpoint being one of the viewpoints.

2. The image processing apparatus according to claim 1, wherein the generating unit generates the viewpoint parallax image and the occlusion parallax image based on a highest correlation disparity value as a disparity value corresponding to the highest one of the correlation values.

3. The image processing apparatus according to claim 2, wherein the generating unit generates the viewpoint parallax image based on the highest correlation disparity value of a first pair among the pairs of viewpoints, and generates the occlusion parallax image based on the highest correlation disparity value of a second pair, the second pair being different from the first pair among the pairs of viewpoints.

4. The image processing apparatus according to claim 3, wherein the generating unit uses a mean value of the highest correlation disparity values of a plurality of first pairs as the disparity value of the viewpoint parallax image, and uses a mean value of the highest correlation disparity values of a plurality of second pairs as the disparity value of the occlusion parallax image, each of the first pairs being the first pair, each of the second pairs being the second pair.

5. The image processing apparatus according to claim 3, wherein the generating unit generates the viewpoint parallax image and the occlusion parallax image based on a table associating a pattern of the correlation values of the respective pairs with the first pair and the second pair.

6. The image processing apparatus according to claim 1, further comprising

a virtual-viewpoint image generating unit configured to generate a captured image of a virtual viewpoint based on the captured images of the viewpoints and the parallax images of the reference viewpoint generated by the generating unit.

7. The image processing apparatus according to claim 6, further comprising

a virtual-focus image generating unit configured to generate a captured image of a virtual focus, using the captured image of the virtual viewpoint gcnerated by the virtual-viewpoint image generating unit and the captured images of the viewpoints.

8. An image processing method comprising:

a calculation step of calculating correlation values of captured images of a plurality of pairs of viewpoints, using captured images of the viewpoints; and

a generation step of generating parallax images of different hierarchical levels of a reference viewpoint based on the correlation values calculated in the calculation step, the parallax images being a viewpoint parallax image formed with a disparity value of the reference viewpoint and an occlusion parallax image formed with a disparity value of an occlusion region, the reference viewpoint being one of the viewpoints,

the calculation step and the generation step being carried out by an image processing apparatus.

9. An image processing apparatus comprising:

a calculating unit configured to calculate correlation values of captured images of a plurality of pairs of viewpoints, using captured images of the viewpoints; and

a generating unit configured to generate a viewpoint parallax image based on a table associating a pattern of the correlation values of the respective pairs with a predetermined pair among the pairs of viewpoints, the viewpoint parallax image being generated from a highest correlation disparity value as a disparity value corresponding to a highest correlation value of the predetermined pair, the viewpoint parallax image being formed with a disparity value of a reference viewpoint, the reference viewpoint being one of the viewpoints.

10. An image processing method comprising:

a calculation step of calculating correlation values of captured images of a plurality of pairs of viewpoints, using captured images of the viewpoints; and

a generation step of generating a viewpoint parallax image based on a table associating a pattern of the correlation values of the respective pairs with a predetermined pair among the pairs of viewpoints, the viewpoint parallax image being generated from a highest correlation disparity value as the disparity value corresponding to the highest correlation value of the predetermined pair, the viewpoint parallax image being formed with a disparity value of a reference viewpoint, the reference viewpoint being one of the viewpoints,

the calculation step and the generation step being carried out by an image processing apparatus.