IMAGE PROCESSING APPARATUS THAT ESTIMATES DISTANCE INFORMATION, METHOD OF CONTROLLING THE SAME, AND STORAGE MEDIUM

• US 20130329019A1
• Filed: 06/05/2013
• Published: 12/12/2013
• Est. Priority Date: 06/07/2012
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Matsuoka, Masaaki
• Granted Patent: US 10,205,928 B2
• Time in Patent Office: Days
• US Class Current: 348/49
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/218 using spatial multiplexing

Abstract

An image processing apparatus free from the inconvenience of an increase in the number of focal positions for image pickup or reduction of the distance accuracy. An optical imaging system forms an object image and an image pickup device picks up the formed object image. A first range image is generated from a plurality of images picked up by the image pickup device and having parallax. A second range image is generated from a plurality of images picked up by the image pickup device and having different degrees of focus at respective corresponding locations therein. A synthesis coefficient is calculated according to the depth of field of a main object selected from objects shown in the object image. A synthesized range image is generated by synthesizing the generated first range image and second range image using the calculated synthesis coefficient.

Claims

1. An image processing apparatus comprising:

an imaging optical unit configured to form an object image;

an image pickup unit configured to pick up an object image formed by said imaging optical unit;

a first range image generation unit configured to generate a first range image from a plurality of images picked up by said image pickup unit and having parallax;

a second range image generation unit configured to generate a second range image from a plurality of images picked up by said image pickup unit and having different degrees of focus at respective corresponding locations therein;

a calculation unit configured to calculate a synthesis coefficient according to the depth of field of a main object selected from a plurality of objects shown in the object image; and

a generation unit configured to generate a synthesized range image by synthesizing the first range image generated by said first range image generation unit and the second range image generated by said second range image generation unit using the synthesis coefficient calculated by said calculation unit.

2. The image processing apparatus according to claim 1, wherein the synthesis coefficient is a coefficient which monotonically increases as the depth of field of the main object increases, and

wherein said generation unit generates, as the synthesized range image, an image in which a value calculated by weighted averaging of respective pixel values of each pixel in the first range image and each pixel in the second range image, which correspond to each other according to location in the first range image and the second range image, is set as a new pixel value, and in the weighted averaging, the synthesis coefficient is a weight coefficient for each pixel value of the first range image.

3. The image processing apparatus according to claim 2, wherein a weight coefficient for each pixel value of the second range image monotonically decreases as the synthesis coefficient increases.

4. The image processing apparatus according to claim 2, wherein said calculation unit calculates the synthesis coefficient such that values taken within a range of the depth of field are larger than values taken outside the range of the depth of field.

5. The image processing apparatus according to claim 2, wherein said calculation unit calculates the synthesis coefficient further using the depth of field of a non-main object different from the main object.

6. The image processing apparatus according to claim 5, wherein said first range image generation unit further generates another first range image associated with the non-main object,

wherein said calculation unit further calculates another synthesis coefficient according to the depth of field of the non-main object, and

wherein said generation unit generates the synthesized range image by further synthesizing an image generated by synthesizing said another first range image and the second range image using said another synthesis coefficient, and the first range image, using the synthesis coefficient.

7. The image processing apparatus according to claim 1, wherein the depth of field is determined by an object distance, and a focal length, an aperture value, and a diameter of a permissible circle of confusion, which are determined by said imaging optical unit and said image pickup unit.

8. The image processing apparatus according to claim 1, wherein the plurality of images picked up by said image pickup unit and having different degrees of focus at respective corresponding locations therein are formed by images which are different in one of a focal position and an aperture value of said imaging optical unit.

9. A method of controlling an image processing apparatus including an imaging optical unit configured to form an object image, and an image pickup unit configured to pick up an object image formed by the imaging optical unit, comprising:

generating a first range image from a plurality of images picked up by the image pickup unit and having parallax;

generating a second range image from a plurality of images picked up by the image pickup unit and having different degrees of focus at respective corresponding locations therein;

calculating a synthesis coefficient according to the depth of field of a main object selected from a plurality of objects shown in the object image; and

generating a synthesized range image by synthesizing the generated first range image and the generated second range image using the calculated synthesis coefficient.

10. A non-transitory computer-readable storage medium storing a computer-executable program for executing a method of controlling an image processing apparatus including an optical imaging unit configured to form an object image, and an image pickup unit configured to pick up an object image formed by the imaging optical unit,

wherein the method comprises:

generating a first range image from a plurality of images picked up by the image pickup unit and having parallax;

generating a second range image from a plurality of images picked up by the image pickup unit and having different degrees of focus at respective corresponding locations therein;

calculating a synthesis coefficient according to depth of field of a main object selected from a plurality of objects shown in the object image; and

generating a synthesized range image by synthesizing the generated first range image and the generated second range image using the calculated synthesis coefficient.