Static object reconstruction method and system

• US 9,830,701 B2
• Filed: 08/09/2016
• Issued: 11/28/2017
• Est. Priority Date: 03/18/2014
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Zhang, Guofeng, Bao, Hujun, Wang, Kangkan, Zhou, Jiong
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US15/232,229
• Publication Number:
  US 20160350904A1
• Time in Patent Office: 476 Days
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/85 Stereo camera calibration

Abstract

Embodiments of the present disclosure disclose a static object reconstruction method and system that are applied to the field of graph and image processing technologies. In the embodiments of the present disclosure, when a static object reconstruction system does not obtain, by means of calculation, an extrinsic camera parameter in a preset time when calculating the extrinsic camera parameter based on a three-dimensional feature point, it indicates that depth data collected by a depth camera is lost or damaged, and a two-dimensional feature point is used to calculate the extrinsic camera parameter to implement alignment of point clouds of a frame of image according to the extrinsic camera parameter. In this way, a two-dimensional feature point and a three-dimensional feature point are mixed, which can implement that a static object can also be successfully reconstructed when depth data collected by a depth camera is lost or damaged.

Claims

1. A static object reconstruction method, comprising:

obtaining a three-dimensional feature point and a two-dimensional feature point in a current frame of image of a static object;

calculating an extrinsic camera parameter of the current frame of image by matching the three-dimensional feature point and reference feature points, wherein the reference feature points are formed by accumulation of feature points in multiple frames of images of the static object;

when the extrinsic camera parameter is not obtained in a preset time when the extrinsic camera parameter of the current frame of image is calculated based on the three-dimensional feature point, calculating the extrinsic camera parameter of the current frame of image by at least one of matching the two-dimensional feature point and a three-dimensional feature point in the reference feature points and matching the two-dimensional feature point and a two-dimensional feature point in a previous frame of image of the current frame of image of the static object; and

transforming, using the calculated extrinsic camera parameter, a point cloud of the current frame of image to a reference coordinate system that comprises the reference feature points to model the static object.

2. The method according to claim 1, wherein matching the three-dimensional feature point and reference feature points and calculating the extrinsic camera parameter of the current frame of image comprises:

selecting, from the reference feature points, a candidate corresponding point at a shortest distance from the three-dimensional feature point in the current frame of image;

selecting candidate corresponding points separately corresponding to a subset of three-dimensional feature points in all three-dimensional feature points in the current frame of image;

calculating a model parameter according to the candidate corresponding points separately corresponding to the subset of three-dimensional feature points;

performing scoring on a model corresponding to the model parameter;

cyclically executing the steps of selecting candidate corresponding points, calculating the model parameter, and performing scoring until the model parameter of the model with a highest score is computed to be used as the extrinsic camera parameter of the current frame of image; and

stopping executing the cyclic steps of selecting candidate corresponding points, calculating the model parameter, and performing scoring when a probability that the subset of three-dimensional feature points and the candidate corresponding points are in an abnormal correspondence that is less than a preset value for consecutive K_s times, when a quantity of times of executing the cyclic steps exceeds a preset value, or when a time for executing the cyclic steps exceeds a preset value.

3. The method according to claim 2, wherein selecting candidate corresponding points separately corresponding to the subset of three-dimensional feature points in all the three-dimensional feature points in the current frame of image comprises:

selecting the subset of three-dimensional feature points from all the three-dimensional feature points in the current frame of image according to a probability that feature points in space areas that are in the previous frame of image and that comprise the static object are in a correct correspondence with the reference feature points; and

selecting, from the multiple candidate corresponding points, a candidate corresponding point of the three-dimensional feature point according to a probability of selecting a candidate corresponding point of the feature point in the previous frame of image.

4. The method according to claim 2, wherein before matching the three-dimensional feature point and reference feature points, the method further comprises:

matching the three-dimensional feature point and the feature points in the previous frame of image of the current frame of image of the static object to obtain an initial extrinsic camera parameter;

using the initial extrinsic camera parameter as a condition in determining the extrinsic camera parameter according to a model score; and

matching the three-dimensional feature point and the reference feature points of the static object to obtain the extrinsic camera parameter of the current frame of image.

5. The method according to claim 1, wherein matching the two-dimensional feature point and the three-dimensional feature point in the reference feature points comprises:

matching the two-dimensional feature point and the three-dimensional feature point in the reference feature points to determine a three-dimensional reference feature point corresponding to the two-dimensional feature point; and

using the a second extrinsic camera parameter as the extrinsic camera parameter of the current frame of image, wherein the second extrinsic camera parameter minimizes a camera pose function in the reference coordinate system, and wherein the camera pose function comprises a correspondence between the two-dimensional feature point and the three-dimensional reference feature point.

6. The method according to claim 1, wherein matching the two-dimensional feature point and the two-dimensional feature point in the previous frame of image of the current frame of image of the static object comprises:

matching the two-dimensional feature point and the two-dimensional feature point in the previous frame of image of the current frame of image of the static object to determine the two-dimensional feature point that is in the previous frame of image and is corresponding to the two-dimensional feature point; and

selecting multiple pairs of corresponding feature points that have depth data at both the two-dimensional feature point in the current frame of image and the two-dimensional feature point that is in the previous frame of image and is corresponding to the two-dimensional feature point.

7. The method according to claim 1, wherein when the extrinsic camera parameter is not obtained in the preset time and when the extrinsic camera parameter of the current frame of image is calculated based on the three-dimensional feature point, the method further comprises:

generating, according to collected two-dimensional data of the multiple frames of images of the static object, a source model that comprises depth data;

transforming the source model to a target model using a transformation matrix; and

completing lost depth data in the current frame of image according to the transformed target model, wherein the target model comprises depth data in the current frame of image of the static object, wherein the transformation matrix is a matrix that minimizes a first energy function, wherein the first energy function comprises a distance item and a smoothness item, wherein the distance item is used to indicate a distance between a vertex in the source model and a corresponding vertex in the target model, and wherein the smoothness item is used to restrict transformation of adjacent vertices.

8. The method according to claim 1, wherein after calculating the extrinsic camera parameter of the current frame of image, the method further comprises:

establishing a correspondence between the three-dimensional feature point in the current frame of image and the reference feature points using the extrinsic camera parameter;

adjusting extrinsic camera parameters of N frames of images of the static object to minimize a second energy function, wherein the second energy function comprises a distance between a three-dimensional feature point in an i^th frame of image and a feature point that is transformed from a corresponding reference feature point and is in a coordinate system of the i^th frame of image, and wherein i is a positive integer ranging from 0 to N; and

transforming, using the calculated extrinsic camera parameter the point cloud of the current frame of image to the reference coordinate system using an extrinsic camera parameter obtained after the adjustment is performed on the calculated extrinsic camera parameter.

9. The method according to claim 8, wherein when the extrinsic camera parameter is calculated based on the two-dimensional feature point and after the calculating the extrinsic camera parameter of the current frame of image, the method further comprises establishing a correspondence between the two-dimensional feature point in the current frame of image and the reference feature points using the extrinsic camera parameter, wherein the second energy function further comprises a distance between a two-dimensional feature point in the i^th frame of image and a feature point that is transformed from a corresponding reference feature point and that is in the coordinate system of the i^th frame of image.

10. The method according to claim 1, wherein after calculating the extrinsic camera parameter of the current frame of image, the method further comprises:

combining the feature points that overlap in the frame of image and the another frame of image when a feature point in a frame of image of the static object overlaps a feature point in another frame of image to obtain updated reference feature points;

updating an extrinsic camera parameter of each frame of image of the static object according to the updated reference feature points; and

transforming, using the calculated extrinsic camera parameter, the point cloud of the current frame of image to the reference coordinate system using an extrinsic camera parameter obtained after the updating is performed on the calculated extrinsic camera parameter.

11. A static object reconstruction system, comprising:

a processor; and

a non-transitory computer-readable storage medium coupled to the processor and storing programming instructions for execution by the processor,

wherein the programming instructions instruct the processor to;

obtain a three-dimensional feature point and a two-dimensional feature point in a current frame of image of a static object;

calculate an extrinsic camera parameter of the current frame of image by matching the three-dimensional feature point and reference feature points, wherein the reference feature points are formed by accumulation of feature points in multiple frames of images of the static object;

when does not obtain, by means of calculation, the extrinsic camera parameter in a preset time when calculating the extrinsic camera parameter of the current frame of image based on the three-dimensional feature point, calculate the extrinsic camera parameter of the current frame of image by at least one of matching the two-dimensional feature point and a three-dimensional feature point in the reference feature points and matching the two-dimensional feature point and a two-dimensional feature point in a previous frame of image of the current frame of image of the static object; and

transform, using the extrinsic camera parameter, a point cloud of the current frame of image to a reference coordinate system that comprises the reference feature points to model the static object.

12. The system according to claim 11, wherein the programming instructions instruct the processor to:

select, from the reference feature points, a candidate corresponding point at a shortest distance from the three-dimensional feature point in the current frame of image;

select candidate corresponding points separately corresponding to a subset three-dimensional feature points in all three-dimensional feature points in the current frame of image;

calculate a model parameter according to the candidate corresponding points that are separately corresponding to the subset of three-dimensional feature points;

perform scoring on a model corresponding to the model parameter; and

use a model parameter of a model with a highest score among scores as the extrinsic camera parameter of the current frame of image in response to cyclically executing the steps of selecting candidate corresponding points, calculating the model parameter, and performing scoring until the model with the highest score is computed; and

stop executing the cyclic steps of selecting candidate corresponding points, calculating the model parameter, and performing scoring when a probability that the subset of three-dimensional feature points and the candidate corresponding points are in an abnormal correspondence is less than a preset value for consecutive times, when a quantity of times of executing the cyclic steps exceeds a preset value, or when a time for executing the cyclic steps exceeds a preset value.

13. The system according to claim 12, wherein the programming instructions instruct the processor to:

select the subset of three-dimensional feature points from all the three-dimensional feature points in the current frame of image according to a probability that feature points in space areas that are in the previous frame of image and that comprise the static object are in a correct correspondence with the reference feature points; and

select, from the multiple candidate corresponding points, a candidate corresponding point of the three-dimensional feature point according to a probability of selecting a candidate corresponding point of the feature point in the previous frame of image.

14. The system according to claim 12, wherein the programming instructions instruct the processor to:

match the three-dimensional feature point and the feature points in the previous frame of image of the current frame of image of the static object to obtain an initial extrinsic camera parameter;

use the initial extrinsic camera parameter as a condition of determining the extrinsic camera parameter according to a model score; and

match the three-dimensional feature point and the reference feature points of the static object to finally obtain the extrinsic camera parameter of the current frame of image.

15. The system according to claim 11, wherein the programming instructions instruct the processor to:

match the two-dimensional feature point and the three-dimensional feature point in the reference feature points to determine a three-dimensional reference feature point corresponding to the two-dimensional feature point;

use determined second extrinsic camera parameter as the extrinsic camera parameter of the current frame of image, wherein the second extrinsic camera parameter minimizes a camera pose function in the reference coordinate system, and wherein the camera pose function comprises a correspondence between the two-dimensional feature point and the three-dimensional reference feature point.

16. The system according to claim 15, wherein the programming instructions instruct the processor to:

match the two-dimensional feature point and the two-dimensional feature point in the previous frame of image of the current frame of image of the static object to determine the two-dimensional feature point that is in the previous frame of image and is corresponding to the two-dimensional feature point; and

select multiple pairs of corresponding feature points that have depth data at both the two-dimensional feature point in the current frame of image and the feature point that is in the previous frame of image and is corresponding to the two-dimensional feature point.

17. The system according to claim 11, wherein the programming instructions further instruct the processor to:

generate, according to collected two-dimensional data of the multiple frames of images of the static object, a source model that comprises depth data when determining that calculating the extrinsic camera parameter of the current frame of image based on the three-dimensional feature point fails, wherein the target model comprises depth data in the current frame of image of the static object, wherein the transformation matrix is a matrix that minimizes a first energy function, wherein the first energy function comprises a distance item and a smoothness item, wherein the distance item is used to indicate a distance between a vertex in the source model and a corresponding vertex in the target model, and wherein the smoothness item is used to restrict transformation of adjacent vertices;

transform, to the target model using the transformation matrix, the source model; and

complete lost depth data in the current frame of image according to the target model.

18. The system according to claim 11, wherein the programming instructions further instruct the processor to:

establish a correspondence between the three-dimensional feature point in the current frame of image and the reference feature points using the extrinsic camera parameter;

adjust extrinsic camera parameters of N frames of images of the static object to minimize a second energy function, wherein the second energy function comprises a distance between a three-dimensional feature point in the i^th frame of image and a feature point that is transformed from a corresponding reference feature point and is in a coordinate system of the i^th frame of image, and wherein i is a positive integer ranging from 0 to N; and

transform the point cloud of the current frame of image to the reference coordinate system using an extrinsic camera parameter that is of the current frame of image.

19. The system according to claim 18, wherein the programming instructions instruct the processor to establish a correspondence between the two-dimensional feature point in the current frame of image and the reference feature points using the extrinsic camera parameter when the extrinsic camera parameter is calculated based on the two-dimensional feature point, wherein the second energy function that is used when adjusting the extrinsic camera parameter further comprises a distance between a two-dimensional feature point in the i^th frame of image and a feature point that is transformed from a corresponding reference feature point and that is in the coordinate system of the i^th frame of image.

20. The system according to claim 11, wherein the programming instructions further instruct the processor to:

combine the feature points that overlap in the frame of image and the another frame of image when a feature point in a frame of image of the static object overlaps a feature point in another frame of image;

obtain updated reference feature points according to the feature points;

update an extrinsic camera parameter of each frame of image of the static object according to the updated reference feature points; and

transform the point cloud of the current frame of image to the reference coordinate system using an extrinsic camera parameter that is of the current frame of image and is obtained by means of updating.