Image processing apparatus, image processing method, image processing program, and image pickup apparatus acquiring a focusing distance from a plurality of images

• US 10,021,290 B2
• Filed: 02/21/2017
• Issued: 07/10/2018
• Est. Priority Date: 05/17/2012
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Takahashi, Takahiro
• Primary Examiner(s): Villecco, John
• Application Number:
  US15/437,690
• Publication Number:
  US 20170163876A1
• Time in Patent Office: 504 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: H04N 23/676 Bracketing for image capture at varying focusing conditions
• CPC Class Codes: H04N 23/959 by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics

Abstract

An image processing apparatus that acquires a focusing distance from a plurality of images each having different degrees of blur, comprises a misalignment detection unit that detects misalignment among the plurality of images; a distance information acquisition unit that acquires, based on the plurality of images, distance information, which is information to indicate a focusing distance in an area of the images; and a reliability information acquisition unit that creates, based on the detected misalignment, reliability information, which is information to indicate reliability of the focusing distance acquired for each area, wherein the distance information acquisition unit changes the acquired distance information based on the reliability information.

Claims

1. An image processing apparatus that acquires depth information from a plurality of images each having different degrees of blur, the image processing apparatus comprising:

a processor; and

a memory storing instructions which, when executed by the processor, cause the information processing apparatus to perform steps comprising;

(1) detecting an amount of misalignment and/or a direction of misalignment among the plurality of images;

(2) acquiring, based on the plurality of images aligned based on the detected amount of misalignment and/or the detected direction of misalignment, depth information for each area in an image of the plurality of images; and

(3) creating reliability information based on the detected amount of misalignment and/or the detected direction of misalignment, the reliability information being information to indicate the reliability of the depth information for each area in an image of the plurality of images.

2. The image processing apparatus according to claim 1, wherein depth information for an area whose reliability is lower than a predetermined value is substituted for by depth information corresponding to another area whose reliability is higher than the predetermined value, instead of the acquired depth information.

3. The image processing apparatus according to claim 1, wherein the acquiring comprises determining a correction area where the brightness or the hue of pixels therein is in a predetermined range, and for an area whose reliability is lower than the predetermined value, uses depth information corresponding to another area belonging to a same correction area to which this area belongs instead of the acquired depth information.

4. The image processing apparatus according to claim 1, wherein the creating comprises acquiring a first difference between misalignment in a target area and average misalignment in surrounding areas of the target area, and sets a lower reliability for the target area compared to a case when the difference between misalignment in a target area and average misalignment in surrounding areas of the target area is a second difference smaller than the first difference.

5. The image processing apparatus according to claim 1, wherein the creating creates the reliability information further based on a frequency characteristic of a focused image or a brightness value of pixels therein among the plurality of images photographed.

6. The image processing apparatus according to claim 5, wherein if a spatial frequency of a target area is a first spatial frequency, the creating sets a lower reliability for the target area compared with a case when the spatial frequency is a second spatial frequency higher than the first spatial frequency.

7. The image processing apparatus according to claim 1, wherein the creating creates the reliability information further based on a difference among the plurality of images photographed.

8. The image processing apparatus according to claim 1, wherein the instructions, when executed by the processor, further cause the information processing apparatus to effect depth of field control that blurs a focused image among the plurality of images photographed, using the acquired depth information.

9. The image processing apparatus according to claim 8, wherein the depth of field control decreases a blurring amount for an area having a first reliability compared with a case when the reliability is a second reliability higher than the first reliability.

10. The image processing apparatus according to claim 1, wherein in the detecting, the amount of misalignment and the direction of misalignment among the plurality of images are detected.

11. An image pickup apparatus comprising:

an imaging unit; and

the image processing apparatus according to claim 1.

12. An image processing method used by an image processing apparatus that acquires depth information from a plurality of images each having different degrees of blur, the method comprising the steps of:

detecting an amount of misalignment and/or a direction of misalignment among the plurality of images;

acquiring, based on the plurality of images aligned based on the detected amount of misalignment and/or the detected direction of misalignment, depth information for each area in an image of the plurality of images; and

creating reliability information based on the detected amount of misalignment and/or the detected direction of misalignment, the reliability information being information to indicate the reliability of the depth information for each area in an image of the plurality of images.

13. The image processing method according to claim 12, further comprising the step of changing the acquired depth information based on the reliability information.

14. The image processing method according to claim 12, wherein in the detecting step, the amount of misalignment and the direction of misalignment are detected among the plurality of images.

15. A non-transitory storage medium recording an image processing program for causing an image processing apparatus, which acquires depth information from a plurality of images each having different degrees of blur, to execute a method comprising the steps of:

detecting an amount of misalignment and/or a direction of misalignment among the plurality of images;

acquiring, based on the plurality of images aligned based on the detected amount of misalignment and/or the detected direction of misalignment, depth information for each area in an image of the plurality of images; and

creating reliability information based on the detected amount of misalignment and/or the detected direction of misalignment, the reliability information being information to indicate the reliability of the depth information for each area in an image of the plurality of images.