Apparatus and method for estimating camera motion using depth information, and augmented reality system

• US 9,304,194 B2
• Filed: 01/19/2012
• Issued: 04/05/2016
• Est. Priority Date: 01/20/2011
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd. (Samsung Group)
• Original Assignee: Samsung Electronics Co. Ltd. (Samsung Group)
• Inventors: Rhee, Seon Min, Lee, Yong Beom, Kim, Sang Wook, Kim, Kyung Hwan, Kim, Do Kyoon
• Primary Examiner(s): Patel, Jayanti K
• Assistant Examiner(s): HUANG, FRANK F
• Application Number:
  US13/353,767
• Publication Number:
  US 20120236119A1
• Time in Patent Office: 1,538 Days
• Field of Search: 348/46
• US Class Current: 1/1
• CPC Class Codes: G01S 11/12 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

Provided is a camera motion estimation method and apparatus that may estimate a motion of a depth camera in real time. The camera motion estimation apparatus may extract an intersection point from plane information of a depth image, calculate a feature point associated with each of planes included in the plane information using the extracted intersection point, and extract a motion of a depth camera providing the depth image using the feature point. Accordingly, in an environment where an illumination environment dynamically varies, or regardless of a texture state within a space, the camera motion estimation apparatus may estimate a camera motion.

Claims

1. An apparatus that estimates camera motion, the apparatus comprising:

an intersection point extractor configured to extract an intersection point of three planes from plane information, based on at least one of planes, intersection lines, and a plane equation, the plane information being acquired from a point sample of a depth camera providing a depth image;

a feature point detector configured to calculate a feature point for each plane included in the plane information, using the extracted intersection point;

a motion extractor configured to extract a motion of the depth camera providing the depth image, using the calculated feature point; and

an intersection line extractor configured to extract an edge by applying a differential operator to the depth image,

wherein the motion extractor is configured to extract the motion of the depth camera providing the depth image, using the extracted edge, and

the intersection point extractor is further configured to

generate a subset including three elements among elements included in a plane set extracted as the plane information,

extract three vectors respectively expressing three plane equations within the subset, and

extract the intersection point using the extracted three vectors.

2. The apparatus of claim 1, further comprising:

a plane extractor configured to extract the plane information from the depth image.

3. The apparatus of claim 2, wherein the plane extractor is further configured to define a plane set as the plane information by applying, to a cubic equation, a point sample acquired from the depth camera.

4. The apparatus of claim 1, further comprising:

an intersection line extractor configured to extract an intersection line between two planes included in the plane information.

5. The apparatus of claim 4, wherein the feature point detector is configured to calculate feature points for each of plane included in the plane information, the feature points being spaced apart by a predetermined interval along the extracted intersection line.

6. The apparatus of claim 5, wherein the feature point detector is further configured to calculate the feature points for each plane by adjusting the predetermined interval.

7. The apparatus of claim 1, wherein the motion extractor is further configured to calculate rotation information or translation information between a first depth camera and a second depth camera using a first feature point extracted from the first depth camera and a second feature point extracted from the second depth camera.

9. The apparatus of claim 1, further comprising:

a pose update unit configured to update the depth camera pose.

24. The apparatus of claim 1, wherein

8. An apparatus that estimates camera motion, the apparatus comprising:

an intersection point extractor configured to

extract a first intersection point of three planes from first plane information, based on at least one of planes, intersection lines, and a plane equation, the first plane information being acquired from a first point sample of a first depth camera providing a first frame, and

extract a second intersection point of the three planes from second plane information, based on at least one of the planes, the intersection lines, and the plane equation, the second plane information being acquired from a second point sample of a second depth camera providing a second frame;

a feature point detector configured to

calculate a first feature point for each plane included in the first plane information, using the first extracted intersection point,and

calculate a second feature point for each plane included in the second plane information, using the second extracted intersection point;

a motion extractor configured to extract a motion between the first depth camera and the second depth camera using the first feature point and the second feature point; and

an intersection line extractor configured to extract an edge by applying a differential operator to the depth image,

wherein the motion extractor is configured to extract the motion of the depth camera providing the depth image, using the extracted edge, and

the intersection point extractor is further configured to

generate a subset including three elements among elements included in a plane set extracted as the plane information,

extract three vectors respectively expressing three plane equation within the subset, and

extract the intersection point using the extracted three vectors.

10. The apparatus of claim 8, further comprising:

a plane extractor configured to

define a first plane set by applying, to a cubic equation, the first point sample acquired from the first depth camera, and

define a second plane set by applying, to the cubic equation, the second point sample acquired from the second depth camera.

11. The apparatus of claim 8, further comprising:

an intersection line extractor configured to extract a first intersection line between two planes included in the first plane information; and

the intersection point extractor configured to

generate a first subset including three elements among elements of a first plane set extracted as the first plane information, using the first intersection line,

extract a first vector expressing three plane equations within the first subset, and

extract the intersection point associated with the first frame using the extracted first vector.

12. The apparatus of claim 11, wherein the feature point detector is configured to calculate first feature points for each plane included in the first plane information, the first feature points being spaced apart by a predetermined interval along the extracted first intersection line.

13. A method of estimating a camera motion, the method comprising:

extracting an intersection point of three planes from plane information, based on at least one of planes, intersection lines, and a plane equation, the plane information being acquired from a point sample of a depth image provided by a depth camera;

calculating a feature point for each plane included in the plane information, using the extracted intersection point; and

extracting a motion of the depth camera providing the depth image, using the calculated feature point; and

extracting an edge by applying a differential operator to the depth image,

wherein the motion of the depth camera providing the depth image is extracted using the extracted edge, and

the intersection point is extracted by

generating a subset including three elements among elements including in a plane set extracted as the plane information,

extracting three vectors respectively expressing three plane equations within the subset, and

extracting the intersection point using the extracted three vectors.

14. The method of claim 13, further comprising:

defining a plane set as the plane information by applying, to a cubic equation, the point sample that is acquired from the depth camera.

15. The method of claim 13, wherein the extracting comprises:

extracting an intersection line between two planes included in the plane information;

generating a subset including i elements using the extracted intersection line, wherein i denotes a number of planes;

extracting vectors expressing i plane equations within the subset; and

configuring i (n×

n) matrices using the extracted vectors, and extracting the intersection point using the i (n×

n) matrices, wherein n denotes a natural number.

16. The method of claim 15, wherein the calculating comprises calculating feature points for each plane included in the plane information, the feature points being spaced apart by a predetermined interval along the extracted intersection line.

17. The method of claim 13, further comprising:

extracting an edge by applying a differential operator to the depth image,

wherein the extracting of the motion comprises extracting the motion of the depth camera providing the depth image, using the extracted edge.

18. An apparatus that estimates camera motion, the apparatus comprising:

an intersection point extractor configured to extract an intersection point of three planes from plane information, based on at least one of planes, an intersection lines, and a plane equation, the plane information being acquired from a point sample of a depth camera providing a depth image;

a feature point detector configured to calculate a feature point for each plane included in the plane information, using the extracted intersection point;

an edge extractor configured to extract an edge by applying a first differential operator to a depth image in response to the plane information being absent from the depth image;

a motion extractor configured to extract a motion of the depth camera providing the depth image, using the calculated feature point or the extracted edge; and

an intersection line extractor configured to extract an edge by applying a differential operator to the depth image,

wherein the motion extractor is configured to extract the motion of the depth camera providing the depth image, using the extracted edge, and

the intersection point extractor is further configured to

generate a subset including three elements among elements including in a plane set extracted as the plane information,

extract three vectors respectively expressing three plane equations within the subset, and

extract the intersection point using the extracted three vectors.

19. The apparatus of claim 18, wherein the edge extractor is further configured to extract the edge by applying a second differential operator to the depth image.

20. The apparatus of claim 18, further comprising:

a depth image capturing unit configured to capture the depth image.

21. The apparatus of claim 18, further comprising:

a pose update unit configured to update a depth camera pose with the extracted motion.

22. The apparatus of claim 18, wherein the differential operator is a gradient operator or a Laplacian operator, and is classified as an x-directional operator or a y-directional operator.

23. The apparatus of claim 18, wherein the edge extractor is further configured to extract the edge by applying a Laplacian operator of applying Laplacian mask, which measures a brightness change amount in the depth image, and extracting a portion of the depth image with a relatively large change.