APPARATUS AND METHOD FOR ESTIMATING CAMERA MOTION USING DEPTH INFORMATION, AND AUGMENTED REALITY SYSTEM

• US 20120236119A1
• Filed: 01/19/2012
• Published: 09/20/2012
• Est. Priority Date: 01/20/2011
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: RHEE, Seon Min, Lee, Yong Beom, Kim, Sang Wook, Kim, Do Kyoon, Kim, Kyung Hwan
• Granted Patent: US 9,304,194 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G01S 11/12 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

Provided is a camera motion estimation method and apparatus that may estimate a motion of a depth camera in real time. The camera motion estimation apparatus may extract an intersection point from plane information of a depth image, calculate a feature point associated with each of planes included in the plane information using the extracted intersection point, and extract a motion of a depth camera providing the depth image using the feature point. Accordingly, in an environment where an illumination environment dynamically varies, or regardless of a texture state within a space, the camera motion estimation apparatus may estimate a camera motion.

Claims

1. An apparatus for estimating a camera motion, the apparatus comprising:

an intersection point extractor to extract an intersection point from plane information of a depth image;

a feature point detector to calculate a feature point associated with each of planes included in the plane information, using the extracted intersection point; and

a motion extractor to extract a motion of a depth camera providing the depth image, using the calculated feature point.

2. The apparatus of claim 1, further comprising:

a plane extractor to extract plane information from the depth image.

3. The apparatus of claim 2, wherein the plane extractor defines a plane set as the plane information by applying, to a cubic equation, a point sample that is acquired from the depth camera.

4. The apparatus of claim 1, wherein the intersection point extractor generates a subset including three elements among elements included in a plane set extracted as the plane information, and extracts three vectors respectively expressing three plane equations within the subset, and extracts the intersection point using the extracted three vectors.

5. The apparatus of claim 1, further comprising:

a intersection line extractor to extract an intersection line between two planes included in the plane information.

6. The apparatus of claim 5, wherein the feature point detector calculates the feature point associated with each of planes spaced apart by a predetermined interval along the extracted intersection line based on the intersection point.

7. The apparatus of claim 6, wherein the feature point detector calculates a plurality of feature points by adjusting the predetermined interval.

8. The apparatus of claim 1, wherein the motion extractor calculates rotation information or translation information between a first depth camera and a second depth camera using a first feature point extracted from the first depth camera and a second feature point extracted from the second depth camera.

9. The apparatus of claim 1, further comprising:

an intersection line extractor to extract an edge by applying a differential operator to the depth image,

wherein the motion extractor extracts the motion of the depth camera providing the depth image, using the extracted edge.

11. The apparatus of claim 1, further comprising:

a pose update unit to update the depth camera pose.

10. An apparatus for estimating a camera motion, the apparatus comprising:

a feature point detector to calculate a first feature point included in first plane information that is extracted from a first frame of a first depth camera, and a second feature point included in second plane information that is extracted from a second frame of a second depth camera; and

a motion extractor to extract a motion between the first depth camera and the second depth camera using the first feature point and the second feature point.

12. The apparatus of claim 10, further comprising:

a plane extractor to define a first plane set by applying, to a cubic equation, a first point sample acquired from the first depth camera, and to define a second plane set by applying, to the cubic equation, a second point sample acquired from the second depth camera.

13. The apparatus of claim 10, further comprising:

an intersection line extractor to extract a first intersection line between two planes included in the first plane information; and

an intersection point extractor to generate a first subset including three elements among elements of a first plane set extracted as the first plane information, using the first intersection line, to extract a first vector expressing three plane equations within the first subset, and to extract an intersection point associated with a first frame using the extracted first vector.

14. The apparatus of claim 13, wherein the feature point detector calculates at least one first feature point associated with each of planes spaced apart by a predetermined interval along the extracted first intersection line based on the first intersection point.

15. A method of estimating a camera motion, the method comprising:

extracting an intersection point from plane information of a depth image that is acquired from a depth camera;

calculating a feature point associated with each of planes included in the plane information, using the extracted intersection point; and

extracting a motion of the depth camera providing the depth image, using the calculated feature point.

16. The method of claim 15, further comprising:

defining a plane set as the plane information by applying, to a cubic equation, a point sample that is acquired from the depth camera.

17. The method of claim 15, wherein the extracting comprises:

extracting an intersection line between two planes included in the plane information;

generating a subset including i elements using the extracted intersection line, wherein i denotes a number of planes;

extracting vectors expressing i plane equations within the subset; and

configuring i (n×

n) matrices using the extracted vectors, and extracting the intersection point using the i (n×

n) matrices, wherein n denotes a natural number.

18. The method of claim 17, wherein the calculating comprises calculating at least one feature point spaced apart by a predetermined interval along the extracted intersection line based on the intersection point.

19. The method of claim 15, further comprising:

extracting an edge by applying a differential operator to the depth image,

wherein the extracting of the motion comprises extracting the motion of the depth camera providing the depth image, using the extracted edge.

20. An augmented reality system, comprising:

a camera motion estimation apparatus to estimate a motion of a depth camera using a depth image;

a pose update unit to update a camera pose with the estimated motion;

a virtual camera pose update unit to update a virtual camera pose using the updated camera pose;

a color image capturing unit to capture a color image from a color camera having the same viewpoint as the depth camera;

a matching unit to match the updated virtual camera pose with a virtual object by referring to a virtual object database; and

a real image/virtual object matching unit to match the captured color image and the matched virtual object.

21. An augmented reality system, comprising:

a depth camera to generate a depth image;

a camera motion estimation apparatus to estimate a motion of the depth camera using the depth image;

a pose update unit to update a camera pose with the estimated motion;

a color image capturing unit to capture a color image from a color camera having a viewpoint different from the depth camera;

a camera calibrator to calibrate different viewpoints of the depth image and the color image so that the depth image and the color image have the same viewpoint;

a virtual camera pose update unit to update a virtual camera pose using the updated camera pose and the calibrated viewpoint;

a matching unit to match the updated virtual camera pose with a virtual object by referring to a virtual object database; and

a real image/virtual object matching unit to match the color image with the calibrated viewpoint and the matched virtual object.

22. An apparatus for estimating a camera motion, the apparatus comprising:

an edge extractor to extract an edge by applying a differential operator to a depth image; and

a motion extractor to extract a motion of a depth image providing the depth image, using the extracted edge.

23. The apparatus of claim 22, wherein the edge extractor extracts the edge by applying a first differential operator or a second differential operator to the depth image.

24. The apparatus of claim 22, further comprising:

a depth image capturing unit to capture the depth image.

25. The apparatus of claim 22, further comprising:

a pose update unit to update a depth camera pose with the extracted motion.