SYSTEM AND METHOD FOR THREE-DIMENSIONAL IMAGE RECONSTRUCTION USING AN ABSOLUTE ORIENTATION SENSOR

• US 20180018787A1
• Filed: 07/18/2017
• Published: 01/18/2018
• Est. Priority Date: 07/18/2016
• Status: Active Grant

• Current Assignee: King Abdullah University of Science and Technology
• Original Assignee: King Abdullah University of Science and Technology
• Inventors: Giancola, Silvio, Ghanem, Bernard Semaan, Schneider, Jens, Wonka, Peter
• Granted Patent: US 10,489,927 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects

Abstract

A three-dimensional image reconstruction system includes an image capture device, an inertial measurement unit (IMU), and an image processor. The image capture device captures image data. The inertial measurement unit (IMU) is affixed to the image capture device and records IMU data associated with the image data. The image processor includes one or more processing units and memory for storing instructions that are executed by the one or more processing units, wherein the image processor receives the image data and the IMU data as inputs and utilizes the IMU data to pre-align the first image and the second image, and wherein the image processor utilizes a registration algorithm to register the pre-aligned first and second images.

Claims

1. A three-dimensional image reconstruction system comprising:

an image capture device that captures image data;

an inertial measurement unit (IMU) affixed to the image capture device that records IMU data that is associated with the image data; and

an image processor that includes one or more processing units and memory for storing instructions that are executed by the one or more processing units, wherein the image processor receives the image data and the IMU data as inputs and utilizes the IMU data to pre-align the image data with a current reconstructed point cloud, and wherein the image processor utilizes a registration algorithm to register the pre-aligned image data with the current reconstructed point cloud.

2. The three-dimensional image reconstruction system of claim 1, wherein the image processor includes a calibration module that generates a transformation value X based on two or more pairs of image data and IMU data.

3. The three-dimensional image reconstruction system of claim 2, wherein the transformation value X is calculated based on the equation M_i^camX=XM_i^IMU ∀

i=1, . . . , N, wherein M_i^cam represents a camera pose measurement, M_i^IMU represents an IMU pose measurement, and N represents the number of poses utilized in conjunction with solving the equation.

4. The three-dimensional image reconstruction system of claim 2, wherein the image processor generates a partial point cloud based on the received image data.

5. The three-dimensional image reconstruction system of claim 4, wherein the image processor utilizes the transformation value X and the IMU data to pre-align the partial point cloud with the current reconstructed point cloud.

6. The three-dimensional image reconstruction system of claim 5, wherein the image processor calculates rigid body motion between the partial point cloud and the current reconstructed point cloud and registers the partial point cloud with the current reconstructed point cloud.

7. The three-dimensional image reconstruction system of claim 6, wherein a rigid body motion distance is iteratively calculated between the partial point cloud and the current reconstructed point cloud until a position of the partial point cloud is found that minimizes a sum of the squared distance between points in the partial point cloud and the current reconstructed point cloud.

8. The three-dimensional image reconstruction system of claim 7, wherein the image processor registers the partial point cloud with the current reconstructed point cloud in response to the minimized sum of the squared distance between points in the partial point cloud and the current reconstructed point cloud.

9. A three-dimensional image reconstruction system comprising:

an image capture device that captures image;

an inertial measurement unit (IMU) affixed to the image capture device that records IMU data associated with image data; and

an image processor that includes one or more processing units and memory for storing instructions that are executed by the one or more processing units, wherein the image processor receives the image data and the IMU data as inputs, and calculates rigid body motion that includes a rotational component R and a translational component t between the image data and the current reconstructed point cloud, wherein the rotational component R of the calculated rigid body motion, but not the translational component t, is constrained by the IMU data and wherein the image processor utilizes a registration algorithm to register the image data with the current reconstructed point cloud.

10. The three-dimensional image reconstruction system of claim 9, wherein the image processor includes a calibration module that generates a transformation value X based on two or more pairs of image data and IMU data.

11. The three-dimensional image reconstruction system of claim 10, wherein the image processor utilizes the IMU data to determine the rotational component R of the rigid body motion.

12. The three-dimensional image reconstruction system of claim 11, wherein the image processor utilizes the rotational component R given by the IMU data and the transformation value X to calculates rigid body motion (R, t) between the partial point cloud and the current reconstructed point cloud, wherein only the translational component is required to be solved by the image processor.

13. A method of reconstructing a three-dimensional image from a plurality of received images, the method comprising:

generating a partial point cloud from a received image;

pre-aligning the partial point cloud with a current reconstructed point cloud based on orientation data provided by an inertial measurement unit (IMU);

calculating rigid body motion between the partial point cloud and the current reconstructed point cloud until a distance between points in the partial point cloud and the current reconstructed point cloud is minimized; and

registering the partial point cloud with the current reconstructed point cloud to update the current reconstructed point cloud.

14. The method of claim 13, further including:

calculating a transformation value X based on two or more pairs of received images and orientation data associated with the received images, wherein the transformation value X is calculated based on the equation M_i^camX=XM_i^IMU ∀

i=1, . . . , N, wherein M_i^cam represents a camera pose measurement, M_i^IMU represents an IMU pose measurement, and N represents the number of poses utilized in conjunction with solving the equation.

15. The method of claim 14, wherein pre-aligning the partial point cloud with the current reconstructed point cloud is based on the orientation data and the transformation value X.