Determining camera motion

• US 7,605,817 B2
• Filed: 11/09/2005
• Issued: 10/20/2009
• Est. Priority Date: 11/09/2005
• Status: Active Grant

• Current Assignee: Medit Corp.
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Rohaly, Janos, Zhang, Tong, Weeks, Steven V.
• Primary Examiner(s): Nguyen; Phu K
• Application Number:
  US11/270,135
• Publication Number:
  US 20070103460A1
• Time in Patent Office: 1,441 Days
• Field of Search: 345/419, 345/473, 345/474, 345/475, 382/154, 382/285, 382/107, 356/12, 348/169
• US Class Current: 345/473
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs

Abstract

Camera motion is determined in a three-dimensional image capture system using a combination of two-dimensional image data and three-dimensional point cloud data available from a stereoscopic, multi-aperture, or similar camera system. More specifically, a rigid transformation of point cloud data between two three-dimensional point clouds may be more efficiently parameterized using point correspondence established between two-dimensional pixels in source images for the three-dimensional point clouds.

Claims

1. A method of operating a three-dimensional imaging system comprising:

receiving a first image set, the first image set including a first plurality of two-dimensional views of a subject from a first position wherein the subject is substantially larger than an image plane of the first image set;

receiving a second image set, the second image set including a second plurality of two-dimensional views of a subject from a second position;

deriving a first point cloud from the first image set;

identifying a number of corresponding pixels between a number of pixels of one of the first plurality of two-dimensional views and a number of pixels of one of the second plurality of two-dimensional views;

deriving a second point cloud from the number of corresponding pixels of the second image set;

determining a motion from the first position to the second position using the first point cloud and the second point cloud; and

repeating steps of receiving the first image set, receiving the second image set, deriving the first point cloud, identifying the number of corresponding pixels, deriving the second point cloud, and determining the motion, for a plurality of consecutive three-dimensional images whereby an aggregate point cloud is constructed for the subject.

2. The method of claim 1 wherein the subject is fixed, the first position and the second position being different positions of a moving camera.

3. The method of claim 1 wherein a camera is fixed and the subject moves, the first position and the second position being different positions of the subject.

4. The method of claim 1 wherein determining motion includes determining a rigid transformation from the first point cloud to the second point cloud.

5. The method of claim 4 wherein determining motion includes weighting one or more points of the point clouds according to confidence.

6. The method of claim 1 wherein the first point cloud is derived from a number of pixels of the first image set using stereo matching.

7. The method of claim 1 wherein the motion includes a change in at least one of a position or an orientation.

8. The method of claim 1 wherein the first point cloud includes a subset of points from the first image set.

9. The method of claim 8 further comprising deriving the subset of points from a sparse mesh of pixels within the first image set.

10. The method of claim 1 wherein the second point cloud includes a subset of points from the second image set.

11. The method of claim 1 wherein one of the plurality of consecutive motions cannot be determined directly from point cloud data of consecutive views, the method further comprising recovering the one of the plurality of consecutive motions through an inverse transformation of one or more next consecutive motions for one or more next consecutive point clouds.

12. The method of claim 1 further comprising supplementing the first point cloud with one or more additional points from the first image set having one or more additional corresponding points in the second image set.

13. The method of claim 1 further comprising supplementing the second point cloud with one or more additional points from the second image set having one or more additional corresponding points in the first image set.

14. The method of claim 1 wherein the first image set and the second image set are obtained from a multi-aperture camera.

15. The method of claim 14 wherein identifying one or more corresponding pixels further includes identifying corresponding pixels between two-dimensional views from a center aperture of the multi-aperture camera.

16. The method of claim 1 wherein the first image set and the second image set are obtained from a camera system including a plurality of cameras in a fixed relationship to one another.

17. The method of claim 1 wherein the subject includes one or more of a nose, an ear, one or more teeth, or a body part.

18. The method of claim 1 wherein the subject includes a dental model.

19. The method of claim 18 wherein the dental model includes one or more of a plaster case, a wax-up, an impression, and a negative impression.

20. The method of claim 1 wherein the subject includes a physical model of an object for use in a digital animation process.

21. A system comprising:

a camera, the camera configured to capture a first image set including a first plurality of two-dimensional views of a subject from a first position and a second image set including a second plurality of two-dimensional views of a subject from a second position; and

a processor coupled in a communicating relationship with the camera, the processor configured to derive a first point cloud from the first image set, identify a number of corresponding pixels between a number of pixels of one of the first plurality of two-dimensional views and a number of pixels of one of the second plurality of two-dimensional views, and derive a second point cloud from the number of corresponding pixels of the second image set, the processor further configured to determine a motion from the first position to the second position using the first point cloud and the second point cloud wherein the subject is substantially larger than an image plane of the first image set, the camera capturing an additional plurality of consecutive image sets and the processor configured to derive a plurality of additional point clouds and determine a plurality of consecutive motions therefrom, whereby an aggregate point cloud is constructed for the subject.

22. The system of claim 21 wherein the subject is fixed, the first position and the second position being different positions of the camera.

23. The system of claim 21 wherein the camera is fixed and the subject moves, the first position and the second position being different positions of the subject.

24. The system of claim 21 wherein the processor is configured to determine motion using a rigid transformation from the first point cloud to the second point cloud.

25. The computer program product of claim 24 wherein the processor is configured to determine motion using weighted values for one or more points of the point clouds.

26. The system of claim 21 wherein the first point cloud is derived from a number of pixels of the first image set using stereo matching.

27. The system of claim 21 wherein the motion includes a change in at least one of a position or an orientation.

28. The system of claim 21 wherein the first point cloud includes a subset of points from the first image set.

29. The system of claim 21 wherein one of the plurality of consecutive motions cannot be determined directly from point cloud data of consecutive image sets, the processor further configured to recover one of the plurality of consecutive motions through an inverse transformation of one or more next consecutive motions for one or more next consecutive point clouds.

30. The system of claim 21 wherein the processor is further configured to supplement the first point cloud with one or more additional points from the first image set having one or more additional corresponding points in the second image set.

31. The system of claim 21 wherein the processor is further configured to supplement the second point cloud with one or more additional points from the second image set having one or more additional corresponding points in the first image set.

32. The system of claim 21 wherein the camera includes a multi-aperture camera.

33. The system of claim 32 wherein the processor is configured to identify corresponding pixels between two-dimensional views from a center aperture of the multi-aperture camera.

34. The system of claim 21 wherein the camera includes a plurality of cameras in a fixed relationship to one another.

35. The system of claim 21 wherein the subject includes one or more of a body part, a nose, an ear, one or more teeth, or a physical model of an object for use in a digital animation process.

36. A system comprising:

first receiving means for receiving a first image set, the first image set including a first plurality of two-dimensional views of a subject from a first position wherein the subject is substantially larger than an image plane of the first image set;

second receiving means for receiving a second image set, the second image including a second plurality of two-dimensional views of a subject from a second position;

deriving means for deriving a first point cloud from the first image set;

first identifying means for identifying a number of corresponding pixels between a number of pixels of one of the first plurality of two-dimensional views and a number of pixels of one of the second plurality of two-dimensional views;

second deriving means for deriving a second point cloud from the number of corresponding pixels of the second image set;

motion determining means for determining a motion from the first position to the second position using the first point cloud and the second point cloud; and

capturing means for capturing an additional plurality of consecutive image sets and deriving a plurality of additional point clouds and determining a plurality of consecutive motions therefrom, whereby an aggregate point cloud is constructed for the subject.