Determining camera motion

• US 20070103460A1
• Filed: 11/09/2005
• Published: 05/10/2007
• Est. Priority Date: 11/09/2005
• Status: Active Grant

• Current Assignee: Medit Corp.
• Original Assignee: 3M Innovative Properties Company (3M Company)
• Inventors: Rohaly, Janos, Zhang, Tong, Weeks, Steven
• Granted Patent: US 7,605,817 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs

Abstract

Camera motion is determined in a three-dimensional image capture system using a combination of two-dimensional image data and three-dimensional point cloud data available from a stereoscopic, multi-aperture, or similar camera system. More specifically, a rigid transformation of point cloud data between two three-dimensional point clouds may be more efficiently parameterized using point correspondence established between two-dimensional pixels in source images for the three-dimensional point clouds.

Claims

1. A method comprising:

receiving a first image set, the first image set including a first plurality of two-dimensional views of a subject from a first position;

receiving a second image set, the second image including a second plurality of two-dimensional views of a subject from a second position;

deriving a first point cloud from the first image set;

identifying a number of corresponding pixels between a number of pixels of one of the first plurality of two-dimensional views and a number of pixels of one of the second plurality of two-dimensional views;

deriving a second point cloud from the number of corresponding pixels of the second image set; and

determining a motion from the first position to the second position using the first point cloud and the second point cloud.

2. The method of claim 1 wherein the subject is fixed, the first position and the second position being different positions of a moving camera.

3. The method of claim 1 wherein a camera is fixed and the subject moves, the first position and the second position being different positions of the subject.

4. The method of claim 1 wherein determining motion includes determining a rigid transformation from the first point cloud to the second point cloud.

5. The method of claim 4 wherein determining motion includes weighting one or more points of the point clouds according to confidence.

6. The method of claim 1 wherein the first point cloud is derived from a number of pixels of the first image set using stereo matching.

7. The method of claim 1 wherein the motion includes a change in at least one of a position or an orientation.

8. The method of claim 1 wherein the first point cloud includes a subset of points from the first image set.

9. The method of claim 8 further comprising deriving the subset of points from a sparse mesh of pixels within the first image set.

10. The method of claim 1 wherein the second point cloud includes a subset of points from the second image set.

11. The method of claim 1 wherein the subject is substantially larger than an image plane of the first image set, the method further comprising repeating the steps of claim 1 for a plurality of consecutive three-dimensional images, whereby an aggregate point cloud is constructed for the subject.

12. The method of claim 11 wherein one of the plurality of consecutive motions cannot be determined directly from point cloud data of consecutive views, the method further comprising recovering the one of the plurality of consecutive motions through an inverse transformation of one or more next consecutive motions for one or more next consecutive point clouds.

13. The method of claim 1 further comprising supplementing the first point cloud with one or more additional points from the first image set having one or more additional corresponding points in the second image set.

14. The method of claim 1 further comprising supplementing the second point cloud with one or more additional points from the second image set having one or more additional corresponding points in the first image set.

15. The method of claim 1 wherein the first image set and the second image set are obtained from a multi-aperture camera.

16. The method of claim 15 wherein identifying one or more corresponding pixels further includes identifying corresponding pixels between two-dimensional views from a center aperture of the multi-aperture camera.

17. The method of claim 1 wherein the first image set and the second image set are obtained from a camera system including a plurality of cameras in a fixed relationship to one another.

18. The method of claim 1 wherein the subject includes one or more of a nose, an ear, one or more teeth, or a body part.

19. The method of claim 1 wherein the subject includes a dental model.

20. The method of claim 19 wherein the dental model includes one or more of a plaster case, a wax-up, an impression, and a negative impression.

21. The method of claim 1 wherein the subject includes a physical model of an object for use in a digital animation process.

22. A computer program product embodied in a computer readable medium comprising computer code which, when executed on a computer, performs the following steps:

receiving a first image set, the first image set including a first plurality of two-dimensional views of a subject from a first position;

receiving a second image set, the second image including a second plurality of two-dimensional views of a subject from a second position;

deriving a first point cloud from the first image set;

identifying a number of corresponding pixels between a number of pixels of one of the first plurality of two-dimensional views and a number of pixels of one of the second plurality of two-dimensional views;

deriving a second point cloud from the number of corresponding pixels of the second image set; and

determining a motion from the first position to the second position using the first point cloud and the second point cloud.

23. The computer program product of claim 22 wherein the subject is fixed, the first position and the second position being different positions of a moving camera.

24. The computer program product of claim 22 wherein a camera is fixed and the subject moves, the first position and the second position being different positions of the subject.

25. The computer program product of claim 22 wherein determining motion includes determining a rigid transformation from the first point cloud to the second point cloud

26. The computer program product of claim 25 wherein determining motion includes weighting one or more points of the point clouds according to confidence.

27. The computer program product of claim 22 wherein the first point cloud is derived from a number of pixels of the first image set using stereo matching.

28. The computer program product of claim 22 wherein the motion includes a change in at least one of a position or an orientation.

29. The computer program product of claim 22 wherein the first point cloud includes a subset of points from the first image set.

30. The computer program product of claim 22 wherein the subject is substantially larger than an image plane of the first image set, the computer program product further comprising code for repeating the steps of claim 22 for a plurality of consecutive three-dimensional images, whereby an aggregate point cloud is constructed for the subject.

31. The computer program product of claim 22 wherein the first image set and the second image set are obtained from a multi-aperture camera.

32. The computer program product of claim 22 wherein the first image set and the second image set are obtained from a camera system including a plurality of cameras in a fixed relationship to one another.

33. The computer program product of claim 22 wherein the subject includes a body part.

34. The computer program product of claim 33 wherein the body part includes one or more of a nose, an ear, a tooth, or a plurality of teeth.

35. The computer program product of claim 22 wherein the subject includes a dental model.

36. A system comprising:

a camera, the camera configured to capture a first image set including a first plurality of two-dimensional views of a subject from a first position and a second image set including a second plurality of two-dimensional views of a subject from a second position; and

a processor coupled in a communicating relationship with the camera, the processor configured to derive a first point cloud from the first image set, identify a number of corresponding pixels between a number of pixels of one of the first plurality of two-dimensional views and a number of pixels of one of the second plurality of two-dimensional views, and derive a second point cloud from the number of corresponding pixels of the second image set, the processor further configured to determine a motion from the first position to the second position using the first point cloud and the second point cloud.

37. The system of claim 36 wherein the subject is fixed, the first position and the second position being different positions of the camera.

38. The system of claim 36 wherein the camera is fixed and the subject moves, the first position and the second position being different positions of the subject.

39. The system of claim 36 wherein the processor is configured to determine motion using a rigid transformation from the first point cloud to the second point cloud

40. The computer program product of claim 39 wherein the processor is configured to determine motion using weighted values for one or more points of the point clouds.

41. The system of claim 36 wherein the first point cloud is derived from a number of pixels of the first image set using stereo matching.

42. The system of claim 36 wherein the motion includes a change in at least one of a position or an orientation.

43. The system of claim 36 wherein the first point cloud includes a subset of points from the first image set.

44. The system of claim 36 wherein the subject is substantially larger than an image plane of the first image set, the camera capturing an additional plurality of consecutive image sets and the processor configured to derive a plurality of additional point clouds and determine a plurality of consecutive motions therefrom, whereby an aggregate point cloud is constructed for the subject.

45. The system of claim 44 wherein one of the plurality of consecutive motions cannot be determined directly from point cloud data of consecutive image sets, the processor further configured to recover one of the plurality of consecutive motions through an inverse transformation of one or more next consecutive motions for one or more next consecutive point clouds.

46. The system of claim 36 wherein the processor is further configured to supplement the first point cloud with one or more additional points from the first image set having one or more additional corresponding points in the second image set.

47. The system of claim 36 wherein the processor is further configured to supplement the second point cloud with one or more additional points from the second image set having one or more additional corresponding points in the first image set.

48. The system of claim 36 wherein the camera includes a multi-aperture camera.

49. The system of claim 48 wherein the processor is configured to identify corresponding pixels between two-dimensional views from a center aperture of the multi-aperture camera.

50. The system of claim 36 wherein the camera includes a plurality of cameras in a fixed relationship to one another.

51. The system of claim 36 wherein the subject includes one or more of a body part, a nose, an ear, one or more teeth, or a physical model of an object for use in a digital animation process.

52. A method comprising determining a motion of a camera from a first position to a second position using a rigid transformation of a point cloud generated from the first position to a point cloud generated from the second position, the point cloud from the second position comprising points selected using a correspondence of two-dimensional image data from the first position and the second position.

53. A system comprising:

first receiving means for receiving a first image set, the first image set including a first plurality of two-dimensional views of a subject from a first position;

second receiving means for receiving a second image set, the second image including a second plurality of two-dimensional views of a subject from a second position;

deriving means for deriving a first point cloud from the first image set;

first identifying means for identifying a number of corresponding pixels between a number of pixels of one of the first plurality of two-dimensional views and a number of pixels of one of the second plurality of two-dimensional views;

second deriving means for deriving a second point cloud from the number of corresponding pixels of the second image set; and

motion determining means for determining a motion from the first position to the second position using the first point cloud and the second point cloud.