System and method for three-dimensional image reconstruction using an absolute orientation sensor

• US 10,489,927 B2
• Filed: 07/18/2017
• Issued: 11/26/2019
• Est. Priority Date: 07/18/2016
• Status: Active Grant

• Current Assignee: King Abdullah University of Science and Technology
• Original Assignee: King Abdullah University of Science and Technology
• Inventors: Giancola, Silvio, Ghanem, Bernard Semaan, Schneider, Jens, Wonka, Peter
• Primary Examiner(s): Wu, Yanna
• Application Number:
  US15/653,163
• Publication Number:
  US 20180018787A1
• Time in Patent Office: 861 Days
• Field of Search: None
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects

Abstract

A three-dimensional image reconstruction system includes an image capture device, an inertial measurement unit (IMU), and an image processor. The image capture device captures image data. The inertial measurement unit (IMU) is affixed to the image capture device and records IMU data associated with the image data. The image processor includes one or more processing units and memory for storing instructions that are executed by the one or more processing units, wherein the image processor receives the image data and the IMU data as inputs and utilizes the IMU data to pre-align the first image and the second image, and wherein the image processor utilizes a registration algorithm to register the pre-aligned first and second images.

Claims

1. A three-dimensional image reconstruction system comprising:

an image capture device that captures image data;

an inertial measurement unit (IMU) affixed to the image capture device that records IMU data that is associated with the image data; and

an image processor that includes one or more processing units and memory for storing instructions that are executed by the one or more processing units, wherein the image processor receives the image data and the IMU data as inputs, wherein the image processor includes a calibration module that generates a transformation value X based on two or more pairs of first image data and two or more pairs of first IMU data during a calibration stage, wherein the image processor subsequently receives second image data and second IMU data and generates a partial point cloud based on the second image data, wherein the image processor utilizes the second IMU data and the transformation value X to pre-align the partial point cloud with a current reconstructed point cloud, and wherein the image processor subsequently utilizes a registration algorithm, without reference to the second IMU data, to register the pre-aligned partial point cloud with the current reconstructed point cloud to generate a reconstructed three-dimensional image.

2. The three-dimensional image reconstruction system of claim 1, wherein the transformation value X is calculated based on an equation M_i^camX=XM_i^IMU ∀

i=1, . . . , N, wherein M_i^cam represents a camera pose measurement, M_i^IMU represents an IMU pose measurement, and N represents a number of poses utilized in conjunction with solving the equation.

3. The three-dimensional image reconstruction system of claim 1, wherein the registration of the pre-aligned partial point cloud with the current reconstructed point cloud by the image processor includes calculating a rigid body motion distance between the partial point cloud and the current reconstructed point cloud and utilizing the calculated rigid body motion distance to register the partial point cloud with the current reconstructed point cloud.

4. The three-dimensional image reconstruction system of claim 3, wherein the rigid body motion distance is iteratively calculated between the partial point cloud and the current reconstructed point cloud until a position of the partial point cloud is found that minimizes a sum of a squared distance between points in the partial point cloud and the current reconstructed point cloud.

5. The three-dimensional image reconstruction system of claim 4, wherein the image processor registers the partial point cloud with the current reconstructed point cloud in response to the minimized sum of the squared distance between the points in the partial point cloud and the current reconstructed point cloud.

6. A method of reconstructing a three-dimensional image from a plurality of received images, the method comprising:

calculating a transformation value X based on two or more pairs of first image data and two or more pairs of first IMU data received from an inertial measurement unit during a calibration stage;

generating a partial point cloud from a second received image;

receiving second IMU data from the inertial measurement unit associated with the second received image;

pre-aligning the partial point cloud with a current reconstructed point cloud based on the second IMU data provided by the inertial measurement unit (IMU) and the transformation value X; and

registering, without reference to the second IMU data, the pre-aligned partial point cloud with the current reconstructed point cloud to generate a reconstructed 3D image, wherein the registration includes calculating a rigid body motion distance between the partial point cloud and the current reconstructed point cloud until the rigid body motion distance between points in the partial point cloud and the current reconstructed point cloud is minimized.

7. The method of claim 6, wherein calculating the transformation value X based on the two or more pairs of first image data and the two or more pairs of first IMU data associated with the two or more pairs of first image data is based on an equation M_i^camX=XM_i^IMU∀

i=1, . . . , N, wherein M_i^cam represents a camera pose measurement, M_i^IMU represents an IMU pose measurement, and N represents a number of poses utilized in conjunction with solving the equation.