METHOD AND APPARATUS FOR FUSING PLURALITY OF DEPTH IMAGES

• US 20180218485A1
• Filed: 03/29/2018
• Published: 08/02/2018
• Est. Priority Date: 09/30/2015
• Status: Abandoned Application

• Current Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V., Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V., Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: XI, Ming, EBEL, Sascha, SCHREER, Oliver, FELDMANN, Ingo
• Application Number:
  US15/940,562
• Publication Number:
  US 20180218485A1
• Time in Patent Office: Days
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/50 Depth or shape recovery

Abstract

The present invention discloses a method and an apparatus for fusing a plurality of depth images. The method includes: obtaining N depth images collected by N image collection units, where N≥2; obtaining a first foreground pixel of an i^th depth image of the N depth images, where i≥1 and i≤N; obtaining N−1 projected points that correspond to the first foreground pixel and that are in depth images respectively collected by N−1 image collection units; and when depth values of a foreground three-dimensional space point in respective coordinate systems of the N−1 image collection units are greater than or equal to depth values of the respective N−1 projected points corresponding to the first foreground pixel, obtaining a three-dimensional point cloud by means of fusion.

Claims

1. A method for fusing a plurality of depth images, wherein the method comprises:

obtaining N depth images collected by N image collection units, wherein N≥

2;

obtaining a first foreground pixel of an i^th depth image of the N depth images, wherein i≥

1 and i≤

N, and the i^th depth image is a depth image collected by an i^th image collection unit of the N image collection units;

back-projecting the first foreground pixel to three-dimensional space, to obtain a foreground three-dimensional space point, and projecting the foreground three-dimensional space point to imaging planes of N−

1 image collection units of the N image collection units other than the i^th collection unit, to obtain N−

1 projected points that correspond to the first foreground pixel and that are in depth images respectively collected by the N−

1 image collection units;

determining whether depth values of the foreground three-dimensional space point in respective coordinate systems of the N−

1 image collection units are greater than or equal to depth values of the respective N−

1 projected points corresponding to the first foreground pixel; and

when the depth values of the foreground three-dimensional space point in the respective coordinate systems of the N−

1 image collection units are greater than or equal to the depth values of the respective N−

1 projected points corresponding to the first foreground pixel, obtaining a three-dimensional point cloud by means of fusion and according to the first foreground pixel and the N−

1 projected points corresponding to the first foreground pixel.

2. The method according to claim 1, wherein the obtaining a three-dimensional point cloud by means of fusion and according to the first foreground pixel and the N−

1 projected points corresponding to the first foreground pixel comprises;

calculating respective depth confidence levels of the N−

1 projected points corresponding to the first foreground pixel, wherein the depth confidence levels of the N−

1 projected points corresponding to the first foreground pixel are respectively used to indicate depth value change degrees of pixels in image areas in which the N−

1 projected points corresponding to the first foreground pixel are respectively located;

determining, according to the respective depth confidence levels of the N−

1 projected points corresponding to the first foreground pixel, a first reliable projected point whose depth confidence level meets a preset condition in the N−

1 projected points corresponding to the first foreground pixel; and

obtaining the three-dimensional point cloud by means of fusion and according to the first foreground pixel and the first reliable projected point.

3. The method according to claim 2, wherein the obtaining the three-dimensional point cloud by means of fusion and according to i first foreground pixel and the first reliable projected point comprises:

calculating an adjacent degree of the first reliable projected point, wherein the adjacent degree of the first reliable projected point is a difference between a depth value of the first reliable projected point and a depth value of the foreground three-dimensional space point, and the depth value of the foreground three-dimensional space point is a depth value of the foreground three-dimensional space point in a coordinate system of an image collection unit used to collect the first reliable projected point;

determining, according the first reliable projected point, a second reliable projected point whose adjacent degree meets a preset condition; and

back-projecting the second reliable projected point to three-dimensional space, to obtain a three-dimensional space point corresponding to the second reliable projected point; and

using three-dimensional coordinate averages of three-dimensional coordinates of the three-dimensional space point corresponding to the second reliable projected point and three-dimensional coordinates of the foreground three-dimensional space point as three-dimensional coordinate values of a three-dimensional space point obtained by means of fusion, and using the three-dimensional space point obtained by means of fusion as a three-dimensional space point of the three-dimensional point cloud obtained by means of fusion, to obtain the three-dimensional point cloud by means of fusion.

4. The method according to claim 3, wherein the method further comprises:

obtaining N color images collected by the N image collection units, wherein the N color images are in a one-to-one correspondence to the N depth images;

obtaining a second color value of a pixel that corresponds to the second reliable projected point and that is in the N color images, and a foreground color value of a pixel that corresponds to the first foreground pixel and that is in the N color images; and

using a color value average of the second color value and the foreground color value as a color value of the three-dimensional space point obtained by means of fusion.

5. The method according to claim 2, wherein the calculating respective depth confidence levels of the N−

1 projected points corresponding to the first foreground pixel comprises;

determining, in an image area in which any projected point of the N−

1 projected points is located, a maximum depth value and a minimum depth value in the image area;

obtaining a difference between the maximum depth value and the minimum depth value, wherein when the difference is greater than a preset difference threshold, the difference is set to be equal to the difference threshold; and

scaling the difference by using a preset scale factor so that the difference is within a preset interval range, and using the scaled difference as a depth confidence level of the projected point.

6. The method according to claim 1, wherein before the determining whether depth values of the foreground three-dimensional space point in respective coordinate systems of the N−

1 image collection units are greater than or equal to depth values of the respective N−

1 projected points corresponding to the first foreground pixel, the method further comprises;

determining whether there is a background pixel in the N−

1 projected points corresponding to the first foreground pixel; and

when there is a background pixel in the N−

1 projected points corresponding to the first foreground pixel, increasing a depth value of the first foreground pixel by a depth offset whose value is within a preset numerical value range, to update the depth value of the first foreground pixel, and performing the determining whether depth values of the foreground three-dimensional space point in respective coordinate systems of the N−

1 image collection units are greater than or equal to depth values of the respective N−

1 projected points corresponding to the first foreground pixel;

or

when there is no background pixel in the N−

1 projected points corresponding to the first foreground pixel, performing the determining whether depth values of the foreground three-dimensional space point in respective coordinate systems of the N−

1 image collection units are greater than or equal to depth values of the respective N−

1 projected points corresponding to the first foreground pixel.

7. The method according to claim 1, wherein the obtaining a first foreground pixel of an i^th depth image of the N depth images comprises:

obtaining a first pixel in the i^th depth image, and when a depth value of the first pixel is less than or equal to a preset depth threshold, and a color value of a pixel that corresponds to the first pixel and that is in an i^th color image is not equal to a background color value of the i^th color image, using the first pixel as the first foreground pixel, wherein the i^th color image is an image collected by the i^th image collection unit.

8. An apparatus for fusing a plurality of depth images, wherein the apparatus comprises:

an image obtaining unit, configured to obtain N depth images collected by N image collection units, wherein N≥

2;

a foreground point obtaining unit, configured to obtain a first foreground pixel of an i^th depth image of the N depth images obtained by the image obtaining unit, wherein i≥

1 and i≤

N, and the i^th depth image is a depth image collected by an i^th image collection unit of the N image collection units;

a projection unit, configured to;

back-project the first foreground pixel obtained by the foreground point obtaining unit to three-dimensional space, to obtain a foreground three-dimensional space point, and project the foreground three-dimensional space point to imaging planes of N−

1 image collection units of the N image collection units other than the i^th collection unit, to obtain N−

1 projected points that correspond to the first foreground pixel and that are in depth images respectively collected by the N−

1 image collection units;

a determining unit, configured to determine whether depth values of the foreground three-dimensional space point obtained by the projection unit in respective coordinate systems of the N−

1 image collection units are greater than or equal to depth values of the respective N−

1 projected points corresponding to the first foreground pixel; and

a fusion unit, configured to;

when the determining unit determines that the depth values of the foreground three-dimensional space point in the respective coordinate systems of the N−

1 image collection units are greater than or equal to the depth values of the respective N−

1 projected points corresponding to the first foreground pixel, obtain a three-dimensional point cloud by means of fusion and according to the first foreground pixel and the N−

1 projected points corresponding to the first foreground pixel.

9. The apparatus according to claim 8, wherein the fusion unit comprises:

a calculation unit, configured to calculate respective depth confidence levels of the N−

1 projected points corresponding to the first foreground pixel, wherein the depth confidence levels of the N−

1 projected points corresponding to the first foreground pixel are respectively used to indicate depth value change degrees of pixels in image areas in which the N−

1 projected points corresponding to the first foreground pixel are respectively located;

a determining unit, configured to determine, according to the respective depth confidence levels of the N−

1 projected points corresponding to the first foreground pixel that are calculated by the calculation unit, a first reliable projected point whose depth confidence level meets a preset condition in the N−

1 projected points corresponding to the first foreground pixel; and

a fusion subunit, configured to obtain the three-dimensional point cloud by means of fusion and according to the first foreground pixel and the first reliable projected point that is determined by the determining unit.

10. The apparatus according to claim 9, wherein the fusion subunit is configured to:

calculate an adjacent degree of the first reliable projected point, wherein the adjacent degree of the first reliable projected point is a difference between a depth value of the first reliable projected point and a depth value of the foreground three-dimensional space point, and the depth value of the foreground three-dimensional space point is a depth value of the foreground three-dimensional space point in a coordinate system of an image collection unit used to collect the first reliable projected point;

determine, according the first reliable projected point, a second reliable projected point whose adjacent degree meets a preset condition;

back-project the second reliable projected point to three-dimensional space, to obtain a three-dimensional space point corresponding to the second reliable projected point; and

use three-dimensional coordinate averages of three-dimensional coordinates of the three-dimensional space point corresponding to the second reliable projected point and three-dimensional coordinates of the foreground three-dimensional space point as three-dimensional coordinate values of a three-dimensional space point obtained by means of fusion, and use the three-dimensional space point obtained by means of fusion as a three-dimensional space point of the three-dimensional point cloud obtained by means of fusion, to obtain the three-dimensional point cloud by means of fusion.

11. The apparatus according to claim 10, wherein the fusion subunit is further configured to:

obtain N color images collected by the N image collection units, wherein the N color images are in a one-to-one correspondence to the N depth images;

obtain a second color value of a pixel that corresponds to the second reliable projected point and that is in the N color images, and a foreground color value of a pixel that corresponds to the first foreground pixel and that is in the N color images; and

use a color value average of the second color value and the foreground color value as a color value of the three-dimensional space point obtained by means of fusion.

12. The apparatus according to claim 9, wherein the calculation unit is configured to:

determine, in an image area in which any projected point of the N−

1 projected points is located, a maximum depth value and a minimum depth value in the image area;

obtain a difference between the maximum depth value and the minimum depth value, wherein when the difference is greater than a preset difference threshold, the difference is set to be equal to the difference threshold; and

scale the difference by using a preset scale factor so that the difference is within a preset interval range, and use the scaled difference as a depth confidence level of the projected point.

13. The apparatus according to claim 8, wherein the determining unit is further configured to:

before determining whether the depth values of the foreground three-dimensional space point in the respective coordinate systems of the N−

1 image collection units are greater than or equal to the depth values of the respective N−

1 projected points corresponding to the first foreground pixel, determine whether there is a background pixel in the N−

1 projected points corresponding to the first foreground pixel; and

when there is a background pixel in the N−

1 projected points corresponding to the first foreground pixel, increase a depth value of the first foreground pixel by a depth offset whose value is within a preset numerical value range, to update the depth value of the first foreground pixel, and perform the determining whether depth values of the foreground three-dimensional space point in respective coordinate systems of the N−

1 image collection units are greater than or equal to depth values of the respective N−

1 projected points corresponding to the first foreground pixel;

or

when there is no background pixel in the N−

1 projected points corresponding to the first foreground pixel, perform the determining whether depth values of the foreground three-dimensional space point in respective coordinate systems of the N−

1 image collection units are greater than or equal to depth values of the respective N−

1 projected points corresponding to the first foreground pixel.

14. The apparatus according to claim 8, wherein the foreground point obtaining unit is configured to:

obtain a first pixel in the i^th depth image, and when a depth value of the first pixel is less than or equal to a preset depth threshold, and a color value of a pixel that corresponds to the first pixel and that is in an i^th color image is not equal to a background color value of the i^th color image, use the first pixel as the first foreground pixel, wherein the i^th color image is an image collected by the i^th image collection unit.