IMAGE PROCESSING APPARATUS, IMAGING APPARATUS AND DISTANCE CORRECTION METHOD

• US 20150139534A1
• Filed: 10/30/2014
• Published: 05/21/2015
• Est. Priority Date: 11/15/2013
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Komatsu, Satoru
• Granted Patent: US 9,799,118 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume

Abstract

An image processing apparatus for correcting, on the basis of an image and a depth map corresponding to the image, the depth map, includes: a detection unit that detects an object included in the image; a determination unit that determines whether a size of the object detected by the detection unit is a threshold or less; and a correction unit that corrects a distance in a target area which corresponds to an area of the object in the depth map, when the size of the detected object is the threshold or less.

Claims

1. An image processing apparatus for correcting a depth map corresponding to an image, comprising:

a detection unit configured to detect an object included in the image;

a determination unit configured to determine whether a size of the object detected by the detection unit is equal to or less than a threshold; and

a correction unit configured to correct a distance in a target area which corresponds to an area of the object in the depth map, when the size of the detected object is equal to or less than the threshold.

2. The image processing apparatus according to claim 1, wherein

the size of the object is at least one of a vertical length, horizontal length or an area of the detected object in the image.

3. The image processing apparatus according to claim 1, wherein

the detection unit is further configured to determine a type of the detected object, and

the determination unit is further configured to change the threshold depending on the type of the detected object.

4. The image processing apparatus according to claim 3, wherein

the determination unit is further configured to decrease the threshold as the object has more texture.

5. The image processing apparatus according to claim 1, wherein

the correction unit is further configured to replaces distances of pixels within a target area corresponding to the area of the object in the depth map with a representative value of the distances in the target area.

6. The image processing apparatus according to claim 5, wherein

the representative value is one of a mean value, a mode and a median of the distances in the target area.

7. The image processing apparatus according to claim 5, wherein

the correction unit is further configured to calculate a mean value of the distances throughout the target area, and to calculate the representative value using only the distances, a difference of which from the mean value is equal to or less than a predetermined value.

8. The image processing apparatus according to claim 5, wherein

the correction unit is further configured to acquire a confidence coefficient of the distance calculation at each position of the depth map, and to calculate the representative value, taking into consideration the confidence coefficient as well.

9. The image processing apparatus according to claim 8, wherein

the correction unit is further configured to calculate the representative value, using only the distances of which confidence coefficient is equal to or greater than a predetermined value.

10. The image processing apparatus according to claim 8, wherein

the correction unit is further configured to calculate the representative value as the weighted mean value of the distances, using the confidence coefficient as the weight.

11. The image processing apparatus according to claim 8, wherein,

the confidence coefficient in the image is calculated as low at pixels where the brightness is equal to or less than a predetermined value, and at pixels where the brightness is saturated.

12. The image processing apparatus according to claim 8, wherein

an entropy or standard deviation is calculated for each local area of the image, and

the confidence coefficient in the image is calculated as low at pixels where the entropy or standard deviation calculated is equal to or less than a predetermined value.

13. The image processing apparatus according to claim 8, wherein

the confidence coefficient is calculated as low at pixels where the difference between the distance information by AF when the image was photographed, and the distance in the depth map, is equal to or more than a predetermined value.

14. The image processing apparatus according to claim 5, further comprising

an area determination unit configured to divide the area of the object into a plurality of divided areas, and to determine each of the plurality of divided areas as the target area, wherein

the correction unit is configured to replace the distances in the target area with a representative value of the distances in the target area, for each of the target areas determined by the area determination unit.

15. The image processing apparatus according to claim 5, further comprising

an area determination unit configured to divide the image into a plurality of divided areas, and to determine a divided area corresponding to the area of the object detected by the detection unit, out of the plurality of divided areas, as the target area, wherein

the correction unit is configured to replace the distances in the target area with a representative value of the distances in the target area, for each of the target areas determined by the area determination unit.

16. The image processing apparatus according to claim 5, further comprising

a verification unit configured to calculate the size of the object based on the size of the object area in the image, an absolute distance to the object acquired by the representative value, and a corresponding absolute distance on the image side, and to determine that an error has occurred when the calculated size is outside a normal range.

17. The image processing apparatus according to claim 1, wherein the object is a human face.

18. The image processing apparatus according to claim 17, wherein the target area is one of a skin color area near a facial area detected by the detection unit, a skin color area and a hair area near the facial area, and divided areas corresponding to the facial area out of the divided areas produced by dividing the image.

19. The image processing apparatus according to claim 1, further comprising

a distance calculation unit configured to generate a depth map based on a plurality of images of different viewpoints, or a plurality of images which are different in the degree of blur, wherein

the image processing apparatus configured to correct the depth map, using at least one of a plurality of images and the depth map.

20. An imaging apparatus, comprising;

an imaging unit; and

the image processing apparatus according to claim 19, wherein

the distance calculation unit generates a depth map based on a plurality of images of different viewpoints or the plurality of images which are different in degree of blur, image-captured by the imaging unit.

21. A distance correction method executed by a computer for correcting a depth map corresponding to an image, comprising:

a detection step of detecting an object included in the image;

a determination step of determining whether a size of the object detected in the detection step is equal to or less than a threshold; and

a correction step of correcting a distance in a target area in the depth map which corresponds to an area of the object, when the size of the detected object is equal to or less than the threshold.

22. A non-transitory computer-readable recording medium recorded with a program allowing a computer to execute each step of the distance correction method according to claim 21.