IMAGE PROCESSING APPARATUS, INTEGRATED CIRCUIT, PROGRAM, IMAGING APPARATUS, AND DISPLAY APPARATUS

• US 20140063295A1
• Filed: 12/26/2012
• Published: 03/06/2014
• Est. Priority Date: 03/15/2012
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Fuchigami, Ikuo, Nakamura, Tsuyoshi, Tanaka, Yoshiteru
• Granted Patent: US 9,002,128 B2
• Time in Patent Office: Days
• US Class Current: 348/239
• CPC Class Codes: G03B 15/00 Special procedures for taking photographs; Apparatus therefor
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 5/002 Denoising; Smoothing noise processing or correction adapted to be used in an image pickup device containing and electronic image sensor H04N23/81, H04N25/60 - H04N25/67
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/958 for extended depth of field imaging
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

An image processing apparatus separates a captured image into a region of object (B) 1001B and a region of object (A) 1001A shallower than the region of object (B) 1001B in depth indicated by depth information, duplicates pixels constituting the region of object (B) 1001B and positioned in neighborhood of boundary between the region of object (B) 1001B and the region of object (A) 1001A, onto the neighborhood of the boundary outside the region of object (B) 1001B, thereby generating an extended region (B) 5001B, performs blur processing on the extended region (B) 5001B and the region of object (A) 1001A based on the depth of the region of object (A) 1001A indicated by the depth information, and after the processing, composites a value of each pixel constituting the extended region (B) 5001B and a value of one of pixels constituting the region of object (A) 1001A corresponding in position.

Claims

1. An image processing apparatus comprising:

a separation unit configured to separate a captured image into a plurality of regions including a first region and a second region that is shallower than the first region in terms of depth indicated by depth information of the captured image;

an extension unit configured to duplicate pixels that constitute the first region and are positioned in a neighborhood of a boundary between the first region and the second region, onto the neighborhood of the boundary outside the first region, thereby generating an extended region;

a blur processing unit configured to perform blur processing on the extended region and the second region based on the depth of the second region indicated by the depth information; and

a composition unit configured to, after the blur processing, composite a value of each of pixels constituting the extended region and a value of one of pixels constituting the second region that correspond in position with each other.

2. The image processing apparatus of claim 1, wherein

the composition unit sets a composition ratio of a value of each of the pixels constituting the extended region to a value of a corresponding one of the pixels constituting the second region, such that the composition ratio is increasingly lower as the pixel constituting the extended region is more distant from the first region.

3. The image processing apparatus of claim 1, wherein

the composition unit performs the composition at a composition ratio that is calculated based on an area ratio, in a range where a blur filter has been applied to each target pixel in the blur processing, of the first region to the second region in the captured image.

4. The image processing apparatus of claim 1, wherein

the separation unit, the extension unit, the blur processing unit, and the composition unit repeatedly perform the separation processing, the extension processing, the blur processing, and the composition processing, respectively, thereby adjusting a blur effect.

5. The image processing apparatus of claim 1, wherein

the blur processing unit repeatedly performs the blur processing, thereby adjusting a blur effect.

6. The image processing apparatus of claim 1, wherein

with respect to one target region among the regions,

the extension unit judges whether the target region is adjacent to other of the regions that is shallower than the target region in terms of depth indicated by the depth information, and

when judging that the target region is adjacent to the other region,

the extension unit duplicates pixels that constitute the target region and are positioned in a neighborhood of a boundary between the target region and the adjacent region, onto the neighborhood of the boundary outside the target region, thereby generating an extended region.

7. An integrated circuit comprising:

a separation unit configured to separate a captured image into a plurality of regions including a first region and a second region that is shallower than the first region in terms of depth indicated by depth information of the captured image;

an extension unit configured to duplicate pixels that constitute the first region and are positioned in a neighborhood of a boundary between the first region and the second region, onto the neighborhood of the boundary outside the first region, thereby generating an extended region;

a blur processing unit configured to perform blur processing on the extended region and the second region based on the depth of the second region indicated by the depth information; and

a composition unit configured to, after the blur processing, composite a value of each of pixels constituting the extended region and a value of one of pixels constituting the second region that correspond in position with each other.

8. A program that causes a computer to perform blur effect processing, the blur effect processing comprising:

a separation step of separating a captured image into a plurality of regions including a first region and a second region that is shallower than the first region in terms of depth indicated by depth information of the captured image;

an extension step of duplicating pixels that constitute the first region and are positioned in a neighborhood of a boundary between the first region and the second region, onto the neighborhood of the boundary outside the first region, thereby generating an extended region;

a blur processing step of performing blur processing on the extended region and the second region based on the depth of the second region indicated by the depth information; and

a composition step of, after the blur processing, compositing a value of each of pixels constituting the extended region and a value of one of pixels constituting the second region that correspond in position with each other.

9. An imaging apparatus comprising:

an imaging unit having a stereo camera;

a display unit;

a memory configured to store therein the program of claim 8 and a left image and a right image captured by the stereo camera;

a DSP configured to perform generation processing of generating a depth image indicating depth information for each pixel based on the captured left and right images stored in the memory, and perform the blur effect processing of claim 8 on at least one of the captured left and right images; and

a CPU configured to control processing of storing the left image and the right image captured by the stereo camera in the memory, control processing of causing the DSP to perform the generation processing and the blur effect processing, and control processing of causing the display unit to display a result of the blur effect processing.

10. An imaging apparatus comprising:

an imaging unit having a monocular camera;

a display unit;

a distance sensor unit configured to measure, for each pixel, a distance to a subject captured by the imaging unit;

a memory configured to store therein the program of claim 8, an image captured by the monocular camera, and the distance measured for each pixel by the distance sensor unit;

a DSP configured to perform generation processing of generating a depth image indicating depth information for each pixel based on the captured image stored in the memory and the measured distance for each pixel, and perform the blur effect processing of claim 8 on the captured image; and

a CPU configured to control processing of storing the left image and the right image captured by the monocular camera in the memory, control processing of causing the DSP to perform the generation processing and the blur effect processing, and control processing of causing the display unit to display a result of the blur effect processing.

11. A display apparatus comprising:

a display unit;

a memory configured to store therein the program of claim 8, a captured image, and a depth image indicating depth information for each pixel of the captured image;

a DSP configured to perform the blur effect processing of claim 8 on the captured image; and

a CPU configured to control processing of causing the DSP to perform the blur effect processing, and control processing of causing the display unit to display a result of the blur effect processing.

12. An image processing apparatus comprising:

a blur processing unit configured to, based on a captured image and a depth image indicating depth information of the captured image, perform blur processing on the captured image, thereby generating a blurred image; and

a repetition unit configured to, based on the generated blurred image and the depth information, repeatedly perform the blur processing on the generated blurred image to generate a new blurred image until a number of repetitions of the blur processing reaches a predetermined number.

13. A program that causes a computer to perform blur effect processing, the blur effect processing comprising:

a blur processing step of, based on a captured image and a depth image indicating depth information of the captured image, performing blur processing on the captured image, thereby generating a blurred image; and

a repetition step of, based on the generated blurred image and the depth information, repeatedly performing the blur processing on the generated blurred image to generate a new blurred image until a number of repetitions of the blur processing reaches a predetermined number.