Image processing apparatus, imaging apparatus and distance correction method

• US 9,799,118 B2
• Filed: 10/30/2014
• Issued: 10/24/2017
• Est. Priority Date: 11/15/2013
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Komatsu, Satoru
• Primary Examiner(s): Roz, Mark
• Application Number:
  US14/528,250
• Publication Number:
  US 20150139534A1
• Time in Patent Office: 1,090 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume

Abstract

An image processing apparatus for correcting, on the basis of an image and a depth map corresponding to the image, the depth map, includes: a detection unit that detects an object included in the image; a determination unit that determines whether a size of the object detected by the detection unit is a threshold or less; and a correction unit that corrects a distance in a target area which corresponds to an area of the object in the depth map, when the size of the detected object is the threshold or less.

Claims

1. An image processing apparatus for correcting a depth map corresponding to an image, the image processing apparatus comprising:

a detection unit configured to detect an object included in the image;

a determination unit configured to determine whether the object detected by the detection unit is smaller than a predetermined size or larger than the predetermined size; and

a correction unit configured to correct distance information in a target area, the target area corresponding to an area of the object in the depth map, when the detected object is smaller than the predetermined size, and to not correct the distance information in the target area when the detected object is larger than the predetermined size,

wherein the correction unit is further configured to acquire a confidence coefficient of the distance information, which indicates an estimated accuracy of the distance information, at each position of the depth map, and to correct the distance information in the target area using only the distance information of which the confidence coefficient is equal to or greater than a predetermined value,

wherein the detection unit is further configured to determine a type of the detected object, and

wherein the determination unit is further configured to change the predetermined size depending on the type of the detected object.

2. The image processing apparatus according to claim 1, wherein the determination by the determination unit is made in accordance with at least one of a vertical length, a horizontal length, or an area of the detected object in the image.

3. The image processing apparatus according to claim 1, wherein the determination unit is further configured to decrease the predetermined size as the object has more texture.

4. The image processing apparatus according to claim 1, wherein the correction unit is further configured to replace distance information of pixels within a target area corresponding to the area of the object in the depth map with a representative value of the distance information in the target area.

5. The image processing apparatus according to claim 4, wherein the representative value is one of a mean value, a mode and a median of the distance information in the target area.

6. The image processing apparatus according to claim 4, wherein the correction unit is further configured to calculate a mean value of the distance information throughout the target area, and to calculate the representative value using only the distance information, a difference of which from the mean value is equal to or less than a predetermined value.

7. The image processing apparatus according to claim 4, wherein the correction unit is further configured to calculate the representative value, using only the distance information of which confidence coefficient is equal to or greater than a predetermined value.

8. The image processing apparatus according to claim 4, wherein the correction unit is further configured to calculate the representative value as the weighted mean value of the distance information, using the confidence coefficient as the weight.

9. The image processing apparatus according to claim 1, wherein the confidence coefficient in the image is calculated as low at pixels where the brightness is equal to or less than a predetermined value, and at pixels where the brightness is saturated.

10. The image processing apparatus according to claim 1, wherein an entropy or standard deviation is calculated for each local area of the image, and

wherein the confidence coefficient in the image is calculated as low at pixels where the entropy or standard deviation calculated is equal to or less than a predetermined value.

11. The image processing apparatus according to claim 1, wherein the confidence coefficient is calculated as low at pixels where the difference between the distance information by AF when the image was photographed, and the distance in the depth map, is equal to or more than a predetermined value.

12. The image processing apparatus according to claim 4, further comprising an area determination unit configured to divide the area of the object into a plurality of divided areas, and to determine each of the plurality of divided areas as the target area,

wherein the correction unit is configured to replace the distance information in the target area with a representative value of the distance information in the target area, for each of the target areas determined by the area determination unit.

13. The image processing apparatus according to claim 4, further comprising an area determination unit configured to divide the image into a plurality of divided areas, and to determine a divided area corresponding to the area of the object detected by the detection unit, out of the plurality of divided areas, as the target area,

wherein the correction unit is configured to replace the distance information in the target area with a representative value of the distance information in the target area, for each of the target areas determined by the area determination unit.

14. The image processing apparatus according to claim 4, further comprising a verification unit configured to calculate a size of the object based on a size of an area of the object in the image and the representative value, and to determine that an error has occurred when the calculated size is outside a normal range.

17. The image processing apparatus according to claim 1, further comprising a distance calculation unit configured to generate a depth map based on a plurality of images of different viewpoints, or a plurality of images which are different in the degree of blur,

wherein the image processing apparatus is configured to correct the depth map, using at least one of a plurality of images and the depth map.

18. An imaging apparatus, comprising:

an imaging unit; and

the image processing apparatus according to claim 17,

wherein the distance calculation unit generates a depth map based on a plurality of images of different viewpoints or the plurality of images which are different in degree of blur, image-captured by the imaging unit.

21. The image processing apparatus according to claim 1, wherein the distance information indicates a distance or a value corresponding to a distance.

22. The image processing apparatus according to claim 1, wherein the confidence coefficient is calculated as low at a pixel which has little texture.

25. The image processing apparatus according to claim 1, wherein the object is a human face.

15. An image processing apparatus for correcting a depth map corresponding to an image, the image processing apparatus comprising:

a detection unit configured to detect a human face in the image;

a determination unit configured to determine whether the human face detected by the detection unit is smaller than a predetermined size or larger than the predetermined size; and

a correction unit configured to correct distance information in a target area that corresponds to an area of the detected human face in the depth map, when the detected human face is smaller than the predetermined size, and not to correct the distance information in the target area when the detected human face is larger than the predetermined size,

wherein the detection unit is further configured to determine a type of the detected human face, and

wherein the determination unit is further configured to change the predetermined size depending on the type of the detected human face.

16. The image processing apparatus according to claim 15, wherein the target area is one of (a) a skin color area near a facial area detected by the detection unit, (b) a skin color area and a hair area near the facial area, and (c) one or more divided areas corresponding to the facial area, in a case that the image is divided into divided areas.

19. A distance correction method executed by a computer for correcting a depth map corresponding to an image, the distance correction method comprising:

a detection step of detecting an object included in the image;

a determination step of determining whether the object detected in the detection step is smaller than a predetermined size or larger than the predetermined size; and

a correction step of correcting distance information in a target area in the depth map that corresponds to an area of the object, when the detected object is smaller than the predetermined size, and not correcting the distance information on the target area when the detected object is larger than the predetermined size,

wherein the correction step comprises acquiring a confidence coefficient of the distance information, which indicates an estimated accuracy of the distance information, at each position of the depth map, and correcting the distance information in the target area using only the distance information of which the confidence coefficient is equal to or greater than a predetermined value,

wherein the detection step comprises determining a type of the detected object, and

wherein the determination step comprises changing the predetermined size depending on the type of the detected object.

20. A non-transitory computer-readable recording medium recorded with a program allowing a computer to execute each step of the distance correction method according to claim 19.

26. The distance correction method according to claim 19, wherein the object is a human face.

23. A distance correction method executed by a computer for correcting a depth map corresponding to an image, the distance correction method comprising:

a detection step of detecting a human face included in the image;

a determination step of determining whether the human face detected in the detection step is smaller than a predetermined size or larger than the predetermined size; and

a correction step of correcting distance information in a target area in the depth map that corresponds to an area of the detected human face, when the detected human face is smaller than the predetermined size, and not correcting the distance information on the target area when the detected human face is larger than the predetermined size,

wherein the determination step comprises determining a type of the detected human face, and

wherein the determination step comprises changing the predetermined size depending on the type of the detected human face.

24. A non-transitory computer-readable recording medium recorded with a program allowing a computer to execute each step of the distance correction method according to claim 23.