Image processing apparatus, image processing method, program, and storage medium

• US 10,582,129 B2
• Filed: 12/21/2017
• Issued: 03/03/2020
• Est. Priority Date: 12/27/2016
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Hamano, Hideyuki, Kanda, Akihiko, Tamaki, Yoshihito
• Primary Examiner(s): Hernandez Hernadez, Nelson D.
• Application Number:
  US15/850,102
• Publication Number:
  US 20180184013A1
• Time in Patent Office: 803 Days
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H01L 27/146 Imager structures
• CPC Class Codes: H01L 27/14603 Special geometry or disposition of pixel-elements, address-lines or gate-electrodes
• CPC Class Codes: H01L 27/14643 Photodiode arrays; MOS imagers
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/672 based on the phase difference signals
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 5/265 Mixing

Abstract

An image processing apparatus is provided that includes an acquisition unit configured to acquire a plurality of viewpoint images having different viewpoint; a calculation unit configured to calculate associated information associated with parallax from the plurality of viewpoint images; an image processing unit configured to generate an output image by performing an image process on image data based on the plurality of viewpoint images; a setting unit configured to set the image process using the image processing unit; and a determination unit configured to acquire the associated information, determine an effect of the image process for each area in the output image, and output the determination result.

Claims

1. An image processing apparatus comprising:

at least one processor executing computer executable instructions or at least one circuit which functions as;

an acquisition unit configured to acquire a plurality of viewpoint images having different viewpoint;

a calculation unit configured to calculate associated information associated with parallax from the plurality of viewpoint images;

an image processing unit configured to generate an output image by performing an image process on image data based on the plurality of viewpoint images;

a setting unit configured to set the image process using the image processing unit; and

a determination unit configured to acquire the associated information, determine an effect of the image process for each area in the output image, and output the determination result; and

a display unit configured to display an image of a characteristic area determined by the determination unit that the effect of the image process is large, or both of the image of the characteristic area and the output image,

wherein the image processing unit sets a parameter using the plurality of viewpoint images and the associated information, generates the output image, and outputs the generated output image to the display unit, and

wherein the display unit distinguishes and displays the characteristic area for each degree of effect of an image process corresponding to the parameter or for each setting value of the parameter.

2. The image processing apparatus according to claim 1, wherein the image processing unit performs a process for combining the plurality of viewpoint images and generating the output image.

3. The image processing apparatus according to claim 1, wherein the associated information includes information calculated using at least One of an image shift amount distribution between the plurality of viewpoint images, a predetermined amount of signal of a spatial frequency band associated with the plurality of viewpoint images, and contrast of the plurality of viewpoint images.

4. The image processing apparatus according to claim 3, wherein the determination unit determines that an area in which an amount of image shift is smaller than a threshold value in the image shift amount distribution, an area in which the amount of signal is larger than a threshold value, or an area in which the contrast is larger than a threshold value is an area having a large effect of the image process.

5. The image processing apparatus according to claim 1, wherein the determination unit determines the effect of the image process by referring to a parameter used for setting of the image process.

6. The image processing apparatus according to claim 1, wherein the at least one processor executing the computer executable instructions or the at least one circuit further function as a decision unit configured to decide the output image from the characteristic area.

7. The image processing apparatus according to claim 1, wherein the at least one processor executing the computer executable instructions or the at least one circuit further function as:

a detection unit configured to detect an object area in the output image; and

a decision unit configured to decide an area of the output image from the object area and the characteristic area,

wherein the display unit displays an area decided by the decision unit.

8. The image processing apparatus according to claim 7, wherein the display unit switches between a method of displaying a frame corresponding to the characteristic area and a method of displaying the area in units of pixels of the output image.

9. The image processing apparatus according to claim 8, wherein the display unit performs a display to notify that the characteristic area is unable to be obtained with respect to an output image for which the characteristic area is unable to be obtained.

10. The image processing apparatus according to claim 7, wherein the display unit changes the characteristic area to be displayed before and after content of the image process is decided.

11. The image processing apparatus according to claim 7, wherein the image processing unit subjects the plurality of viewpoint images to a sharpening or smoothing process before the image process.

12. The image processing apparatus according to claim 1, wherein the acquisition unit acquires the plurality of viewpoint images generated from pixel signals obtained by photoelectrically converting light passing through different pupil areas of an image-forming optical system using a plurality of photoelectric conversion units.

13. An imaging apparatus comprising:

the image processing apparatus according to claim 1; and

an imaging element configured to capture an object.

14. The imaging apparatus according to claim 13, wherein the imaging element includes plurality of microlenses and a plurality of photoelectric conversion units and each of the microlenses corresponds to the plurality of photoelectric conversion units, and

signals of the plurality of viewpoint images are generated from signals output from the plurality of photoelectric conversion units corresponding to each microlens.

15. An image processing method comprising:

acquiring a plurality of viewpoint images having different viewpoint;

setting an image process using an image processing unit;

calculating associated information associated with parallax from the plurality of viewpoint images;

generating an output image by performing, by the image processing unit, image processing on image data based on the plurality of viewpoint images;

acquiring the associated information, determining an effect of the image process for each area in the output image, and outputting the determination result;

displaying, by a display unit, an image of a characteristic area determined by determination unit that the effect of the image process is large, or both of the image of the characteristic area and the output image;

setting, by the image processing unit, a parameter using the plurality of viewpoint images and the associated information, generating the output image, and outputting the generated output image to the display unit; and

distinguishing and displaying, by the display unit, the characteristic area for each degree of effect of an image process corresponding to the parameter or for each setting value of the parameter.

16. A non-transitory readable storage medium which stores a program causing a computer to execute an image processing method, the method comprising:

acquiring a plurality of viewpoint images having different viewpoint;

setting an image process using an image processing unit;

calculating associated information associated with parallax from the plurality of viewpoint images;

generating an output image by performing, by the image processing unit, image processing on image data based on the plurality of viewpoint images;

acquiring the associated information, determining an effect of the image process for each area in the output image, and outputting the determination result;

displaying, by a display unit, an image of a characteristic area determined by a determination unit that the effect of the image process is large, or both of the image of the characteristic area and the output image;

setting, by the image processing unit, a parameter using the plurality of viewpoint images and the associated information, generating the output image, and outputting the generated output image to the display unit; and

distinguishing and displaying, by the display unit, the characteristic area for each degree of effect of an image process corresponding to the parameter or for each setting value of the parameter.