IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM

• US 20130215107A1
• Filed: 02/07/2013
• Published: 08/22/2013
• Est. Priority Date: 02/17/2012
• Status: Abandoned Application

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kimura, Seiji, Kobayashi, Goh
• Application Number:
  US13/761,910
• Publication Number:
  US 20130215107A1
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

There is provided an image processing apparatus including a depth map information estimation unit configured to estimate a depth in region units of a two-dimensional image and generate depth map information in which a depth estimation value in region units of a two-dimensional image is set, a reliability information generation unit configured to generate a depth map information reliability by determining a reliability of the depth value set in the depth map information, a depth map information correction unit configured to generate corrected depth map information by correcting the depth map information based on the depth map information reliability, and a 3D image generation unit configured to generate from the two-dimensional image an image for a left eye (L image) and an image for a right eye (R image) to be applied in a three-dimensional image display by applying the corrected depth map information.

Claims

1. An image processing apparatus comprising:

a depth map information estimation unit configured to estimate a depth in region units of a two-dimensional image and generate depth map information in which a depth estimation value in region units of a two-dimensional image is set;

a reliability information generation unit configured to generate a depth map information reliability by determining a reliability of the depth value set in the depth map information;

a depth map information correction unit configured to generate corrected depth map information by correcting the depth map information based on the depth map information reliability; and

a 3D image generation unit configured to generate from the two-dimensional image an image for a left eye (L image) and an image for a right eye (R image) to be applied in a three-dimensional image display by applying the corrected depth map information,

wherein the depth map information estimation unit is configured to

calculate a middle-low region component energy share from a middle-low region component energy and an AC component energy in region units by performing a frequency component analysis in region units of the two-dimensional image, and

generate depth map information in which a depth estimation value is set based on a value of the calculated middle-low region component energy share.

2. The image processing apparatus according to claim 1, wherein the depth map information estimation unit is configured to generate depth map information in which

a depth estimation value indicating a deep (far away) position is set for a region in which the middle-low region component energy share is large, and

a depth estimation value indicating a near (close) position is set for a region in which the middle-low region component energy share is small.

3. The image processing apparatus according to claim 1, wherein the depth map information estimation unit is configured to generate depth map information in which a depth estimation value is set based on a value of the calculated middle-low region component energy share by calculating the middle-low region component energy share from the middle-low region component energy and the AC component energy in region units based on the following formula.

Middle-low region component energy share=(Middle-low region component energy)/(AC component energy)

4. The image processing apparatus according to claim 1, wherein the reliability information generation unit is configured to generate a statistical information reliability calculated by applying peak position information in a depth information histogram, which is frequency distribution information about depth values in the depth map information.

5. The image processing apparatus according to claim 1, wherein the reliability information generation unit is configured to calculate a frequency ratio (MF/PF) between a frequency (PF) of a peak value in a depth information histogram, which is frequency distribution information about depth values in the depth map information, and a frequency (MF) of a local minimum in the depth information histogram, and generate a statistical information reliability based on the frequency ratio (MF/PF).

6. The image processing apparatus according to claim 1, wherein the reliability information generation unit is configured to generate a spatial distribution reliability calculated by applying difference information about depth values in predetermined region units of the depth map information.

7. The image processing apparatus according to claim 1, wherein the reliability information generation unit is configured to generate a luminance reliability, which is a reliability based on a luminance of the two-dimensional image.

8. The image processing apparatus according to claim 1, wherein the reliability information generation unit is configured to generate an external block correspondence reliability by applying an externally-input external block detection signal.

9. The image processing apparatus according to claim 8,

wherein the external block detection signal is at least one of a noise amount measurement result, a signal band measurement result, a face detection result, a telop detection result, EPG information, camera shooting information, or a motion detection result, and

wherein the reliability information generation unit is configured to generate the external block correspondence reliability by applying any of the detection signals.

10. The image processing apparatus according to claim 1, wherein the depth map information correction unit is configured to generate corrected depth map information by determining a blend ratio between the depth map information and a fixed depth map that has a fixed value as a depth value based on the depth map information reliability, and executing blend processing between the depth map information and the fixed depth map by applying the determined bland ratio.

11. The image processing apparatus according to claim 10, wherein the depth map information correction unit is configured to generate corrected depth map information by executing blend processing which, when the depth map information reliability is high, increases a blend ratio of the depth map information, and when the depth map information reliability is low, decreases a blend ratio of the depth map information.

12. The image processing apparatus according to claim 1, wherein the depth map information correction unit is configured to generate corrected depth map information in which a depth value range set in the depth map information is controlled based on the depth map information reliability.

13. The image processing apparatus according to claim 12, wherein the depth map information correction unit is configured to generate corrected depth map information by executing a control which, when the depth map information reliability is high, decreases a contraction width of a depth value range set in the depth map information, and when the depth map information reliability is low, increases a contraction width of a depth value range set in the depth map information.

14. An image processing method executed in an image processing apparatus, the method comprising:

estimating with a depth map information estimation unit a depth in region units of a two-dimensional image and generating depth map information in which a depth estimation value in region units of a two-dimensional image is set;

generating with a reliability information generation unit a depth map information reliability by determining a reliability of the depth value set in the depth map f of a two-dimensional image and information;

generating with a depth map information correction unit corrected depth map information by correcting the depth map information based on the depth map information reliability; and

generating with a 3D image generation unit from the two-dimensional image an image for a left eye (L image) and an image for a right eye (R image) to be applied in a three-dimensional image display by applying the corrected depth map information,

wherein in the depth map information estimation step,

a middle-low region component energy share is calculated from a middle-low region component energy and an AC component energy in region units by performing a frequency component analysis in region units of the two-dimensional image, and

depth map information is generated in which a depth estimation value is set based on a value of the calculated middle-low region component energy share.

15. A program which causes an image processing apparatus to execute image processing, wherein the program is configured to:

estimate in a depth map information estimation unit a depth in region units of a two-dimensional image and generate depth map information in which a depth estimation value in region units of a two-dimensional image is set;

generate in a reliability information generation unit a depth map information reliability by determining a reliability of the depth value set in the depth map f of a two-dimensional image and information;

generate in a depth map information correction unit corrected depth map information by correcting the depth map information based on the depth map information reliability; and

generate in a 3D image generation unit from the two-dimensional image an image for a left eye (L image) and an image for a right eye (R image) to be applied in a three-dimensional image display by applying the corrected depth map information,

wherein in the depth map information estimation,

a middle-low region component energy share is calculated from a middle-low region component energy and an AC component energy in region units by performing a frequency component analysis in region units of the two-dimensional image, and

depth map information is generated in which a depth estimation value is set based on a value of the calculated middle-low region component energy share.