METHOD AND APPARATUS FOR ALIGNING VIDEO TO THREE-DIMENSIONAL POINT CLOUDS

• US 20070031064A1
• Filed: 06/10/2005
• Published: 02/08/2007
• Est. Priority Date: 06/10/2004
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Hsu, Stephen, Nister, David, Zhao, Wenyi
• Granted Patent: US 7,187,809 B2
• Time in Patent Office: Days
• US Class Current: 382/285
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces

Abstract

A method and apparatus for performing two-dimensional video alignment onto three-dimensional point clouds. The system recovers camera pose from camera video, determines a depth map, converts the depth map to a Euclidean video point cloud, and registers two-dimensional video to the three-dimensional point clouds.

Claims

1. A method of processing video comprising:

estimating camera pose from two-dimensional video;

computing motion stereo for the two-dimensional video using the pose estimate to create a three-dimensional video point cloud;

registering three-dimensional sensor data to the three-dimensional video point cloud; and

producing a three-dimensional model from the registered three-dimensional data and the video point cloud.

2. The method of claim 1 wherein the estimating step further comprises:

applying a feature-based matching process to generate local pose in a local coordinate system;

translating the local pose into the pose estimate within a global coordinate system.

3. The method of claim 2 further comprising translating the local pose estimate to a global coordinate system using information received from a satellite navigation system or calibration from two video frames where 3D information of image features are known.

4. The method of claim 1 wherein the motion stereo computation comprises plane-plus-parallax processing.

5. The method of claim 4 further comprising converting a depth map produced by the plane-plus-parallax processing into a point cloud.

6. The method of claim 1 the registering step further comprises:

performing an ICP alignment.

7. The method of claim 6 wherein the registering step further comprises performing a bundle adjustment using aligned key frames within the two dimensional video.

8. The method of claim 1 wherein the motion stereo is a color segmentation stereo algorithm.

9. The method of claim 1 further comprising converting the three-dimensional sensor data into a second point cloud that is matched to the three-dimensional point cloud representing the two-dimensional video to create a composite point cloud.

10. Apparatus for processing video images comprising:

means for estimating camera pose from two-dimensional video;

means for computing motion stereo for the two-dimensional video using the pose estimate to create a three-dimensional point cloud;

means for registering three-dimensional sensor data to the three-dimensional video point cloud; and

means for producing a three-dimensional model from the matched three-dimensional data and the video point cloud.

11. The apparatus of claim 10 wherein the means for estimating further comprises:

means for applying a feature-based matching process to generate local pose in a local coordinate system;

means for translating the local pose into the pose estimate within a global coordinate system.

12. The apparatus of claim 11 further comprising means for translating the pose estimate using information received from a satellite navigation system or calibration from two video frames where 3D information of image features are known.

13. The apparatus of claim 10 wherein the motion stereo computing means comprises means for performing plane-plus-parallax processing.

14. The apparatus of claim 13 further comprising means for converting a depth map produced by the means for performing plane-plus-parallax processing into a point cloud.

15. The apparatus of claim 10 the means for registering further comprises:

means for performing an ICP alignment.

16. The apparatus of claim 15 wherein the means for registering further comprises:

means for performing a bundle adjustment using key frames within the two dimensional video.

17. The apparatus of claim 10 wherein the motion stereo is a color segmentation stereo algorithm.

18. The apparatus of claim 10 further comprising means for converting the three-dimensional sensor data into a second point cloud that is matched to the three-dimensional point cloud representing the two-dimensional video to create a composite point cloud.