Image processing apparatus and image processing method

• US 10,027,947 B2
• Filed: 05/23/2014
• Issued: 07/17/2018
• Est. Priority Date: 06/05/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Ito, Katsuhisa, Hayasaka, Kengo, Mori, Hironori
• Primary Examiner(s): Chowdhury, Nigar
• Application Number:
  US14/893,754
• Publication Number:
  US 20160105660A1
• Time in Patent Office: 1,516 Days
• Field of Search: 348 42, 348 47- 48
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/225 using parallax barriers
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

The present disclosure relates to an image processing apparatus and an image processing method that enable high-precision generation of parallax images of viewpoints with a small amount of calculation.

A correlation generating unit calculates correlation values of captured images of pairs of viewpoints, using captured images of viewpoints. A generating unit generates parallax images of different hierarchical levels of a reference viewpoint based on the correlation values. The parallax images are a parallax image formed with the disparity value of the reference viewpoint, and a parallax image formed with the disparity value of an occlusion region. The reference viewpoint is one of the viewpoints. The present disclosure can be applied to the image processing apparatus of a light field camera, for example.

Claims

1. An image processing apparatus comprising:

a calculating unit configured to calculate correlation values of captured images of a plurality of pairs of viewpoints, each pair being formed between respective viewpoints of a plurality of viewpoints using captured images of the respective viewpoints obtained by an imaging array including each viewpoint of the plurality of viewpoints; and

a generating unit configured to generate a plurality of parallax images of different hierarchical levels of a reference viewpoint based on highest correlation disparity values among the plurality of pairs of viewpoints using the correlation values calculated by the calculating unit, the plurality of parallax images including a viewpoint parallax image formed with a disparity value of the reference viewpoint and an occlusion parallax image formed with a disparity value of an occlusion region, the reference viewpoint being one of the plurality of viewpoints viewing a region other than the occlusion region,

wherein the calculating unit and the generating unit are each implemented via at least one processor.

2. The image processing apparatus according to claim 1, wherein the generating unit generates the viewpoint parallax image based on the highest correlation disparity value of a first pair among the plurality of pairs of viewpoints, and generates the occlusion parallax image based on the highest correlation disparity value of a second pair, the second pair being different from the first pair among the plurality of pairs of viewpoints.

3. The image processing apparatus according to claim 2, wherein the generating unit uses a mean value of the highest correlation disparity values of a plurality of first pairs as the disparity value of the viewpoint parallax image, and uses a mean value of the highest correlation disparity values of a plurality of second pairs as the disparity value of the occlusion parallax image, each of the first pairs being the first pair, each of the second pairs being the second pair.

4. The image processing apparatus according to claim 2, wherein the generating unit generates the viewpoint parallax image and the occlusion parallax image based on a table associating a pattern of the correlation values of the respective pairs with the first pair and the second pair.

5. The image processing apparatus according to claim 1, further comprising:

a virtual-viewpoint image generating unit configured to generate a captured image of a virtual viewpoint based on the captured images of the respective viewpoints and the parallax images of the reference viewpoint generated by the generating unit,

wherein the virtual-viewpoint image generating unit is implemented via at least one processor.

6. The image processing apparatus according to claim 5, further comprising:

a virtual-focus image generating unit configured to generate a captured image of a virtual focus, using the captured image of the virtual viewpoint generated by the virtual-viewpoint image generating unit and the captured images of the respective viewpoints,

wherein the virtual-focus image generating unit is implemented via at least one processor.

7. An image processing method, implemented via a processor, the method comprising:

calculating correlation values of captured images of a plurality of pairs of viewpoints, each pair being formed between respective viewpoints of a plurality of viewpoints using captured images of the viewpoints; and

generating a plurality of parallax images of different hierarchical levels of a reference viewpoint based on highest correlation disparity values among the plurality of pairs of viewpoints using the calculated correlation values, the plurality of parallax images including a viewpoint parallax image formed with a disparity value of the reference viewpoint and an occlusion parallax image formed with a disparity value of an occlusion region, the reference viewpoint being one of the plurality of viewpoints viewing a region other than the occlusion region.

8. An image processing apparatus comprising:

a calculating unit configured to calculate correlation values of captured images of a plurality of pairs of viewpoints, each pair being formed between respective viewpoints of a plurality of viewpoints using captured images of the respective viewpoints obtained by an imaging array including each viewpoint of the plurality of viewpoints; and

a generating unit configured to generate a viewpoint parallax image based on a table associating a pattern of the correlation values of respective pairs with a predetermined pair among the plurality of pairs of viewpoints, the viewpoint parallax image being generated from a highest correlation disparity value as a disparity value corresponding to a highest correlation value of the predetermined pair, the viewpoint parallax image being formed with a disparity value of a reference viewpoint, the reference viewpoint being one of the plurality of viewpoints,

wherein the calculating unit and the generating unit are each implemented via at least one processor.

9. An image processing method, implemented via a processor, the method comprising:

calculating correlation values of captured images of a plurality of pairs of viewpoints, each pair being formed between respective viewpoints of a plurality of viewpoints using captured images of the respective viewpoints obtained by an imaging array including each viewpoint of the plurality of viewpoints; and

generating a viewpoint parallax image based on a table associating a pattern of the correlation values of respective pairs with a predetermined pair among the pairs of viewpoints, the viewpoint parallax image being generated from a highest correlation disparity value as the disparity value corresponding to the highest correlation value of the predetermined pair, the viewpoint parallax image being formed with a disparity value of a reference viewpoint, the reference viewpoint being one of the plurality of viewpoints.