Camera tracking apparatus and method using reconstruction segments and volumetric surface

• US 9,489,740 B2
• Filed: 06/25/2014
• Issued: 11/08/2016
• Est. Priority Date: 11/20/2013
• Status: Expired due to Fees

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Kim, Young Hee, Kim, Jin Ho, Kim, Ki Hong, Lee, Gil Haeng
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US14/315,123
• Publication Number:
  US 20150139532A1
• Time in Patent Office: 867 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 7/579 from motion

Abstract

Provided are an apparatus and method for tracking a camera that reconstructs a real environment in three dimensions by using reconstruction segments and a volumetric surface. The camera tracking apparatus using reconstruction segments and a volumetric surface includes a reconstruction segment division unit configured to divide three-dimensional space reconstruction segments extracted from an image acquired by a camera, a transformation matrix generation unit configured to generate a transformation matrix for at least one reconstruction segment among the reconstruction segments obtained by the reconstruction segment division unit, and a reconstruction segment connection unit configured to rotate or move the at least one reconstruction segment according to the transformation matrix generated by the reconstruction segment division unit and connect the rotated and moved reconstruction segment with another reconstruction segment.

Claims

1. A camera tracking apparatus comprising:

a reconstruction segment generation unit to generate, using a processor, three-dimensional space reconstruction segments for a reconstruction process from an image including depth information obtained by a camera, the reconstruction segments including first and second reconstruction segments;

a transformation matrix generation unit to generate, using a processor, a transformation matrix for the first reconstruction segment having a distortion among the generated reconstruction segments during the reconstruction process; and

a reconstruction segment connection unit to rotate or move, or rotate and move, using a processor, the first reconstruction segment according to the transformation matrix, and to connect the rotated or moved, or the rotated and moved first reconstruction segment with the second reconstruction segment having substantially no distortion,

wherein the reconstruction segment generation unit generates a corresponding one of the reconstruction segments based on a timing when a movement factor of the camera is greater than a preset value.

2. The camera tracking apparatus of claim 1, wherein the transformation matrix generation unit sets a reference camera at a position between a first position of the camera corresponding to a start frame of the first reconstruction segment and a second position of the camera corresponding to an end frame of the second reconstruction segment, and generates a map set for the first reconstruction segment and the second reconstruction segment using the reference camera.

3. The camera tracking apparatus of claim 2, wherein the transformation matrix generation unit generates the map set including a vertex map and a normal map for the first reconstruction segment and the second reconstruction segment using the reference camera, and applies an Iterative Closest Point (ICP) algorithm to the generated map set to generate the transformation matrix of the first reconstruction segment.

4. The camera tracking apparatus of claim 3, wherein the reconstruction segment connection unit applies an inverse matrix of the transformation matrix generated by the transformation matrix generation unit to a voxel grid domain of the first reconstruction segment.

5. The camera tracking apparatus of claim 4, wherein the reconstruction segment connection unit applies the inverse matrix including a rotation matrix and a movement matrix to a signed distance function (SDF) value of the first reconstruction segment.

6. The camera tracking apparatus of claim 5, wherein the reconstruction segment connection unit connects the first reconstruction segment to which the inverse matrix has been applied and the second reconstruction segment based on the SDF value of the first reconstruction segment, an SDF value of the second reconstruction segment, and weight functions of the first reconstruction segment and the second reconstruction segment.

18. The camera tracking apparatus of claim 1, wherein the camera is configured to move along a three-dimensional surface.

7. A camera tracking method comprising:

generating three-dimensional space reconstruction segments for a reconstruction process from an image including depth information obtained by a camera, the reconstruction segments including first and second reconstruction segments;

generating a transformation matrix for the first reconstruction segment having a distortion among the generated reconstruction segments during the reconstruction process;

converting the first reconstruction segment according to the transformation matrix; and

connecting the converted first reconstruction segment with the second reconstruction segment having substantially no distortion,

wherein generating the three-dimensional space reconstruction segments comprises generating a corresponding one of the reconstruction segments based on a timing when a movement of the camera is greater than a preset value.

8. The camera tracking method of claim 7, wherein generating the transformation matrix comprises generating a map set for the first reconstruction segment and the second reconstruction segment based on a position of a reference camera, and

wherein the reference camera obtains an image at the position between a first position of the camera corresponding to a start frame of the first reconstruction segment and a second position of the camera corresponding to an end frame of the second reconstruction segment along a moving route of the camera.

9. The camera tracking method of claim 8, wherein the generated map set includes a vertex map and a normal map for the first reconstruction segment and the second reconstruction segment, and

wherein generating the transformation matrix further comprises applying an Iterative Closest Point (ICP) algorithm to the generated map set to generate the transformation matrix.

10. The camera tracking method of claim 9, wherein converting the first reconstruction segment comprises applying an inverse matrix of the transformation matrix to a voxel grid domain of the first reconstruction segment.

11. The camera tracking method of claim 10, wherein the inverse matrix includes a rotation matrix and a movement matrix, and the inverse matrix is applied to a signed distance function (SDF) value of the first reconstruction segment.

12. The camera tracking method of claim 11, wherein connecting the converted first reconstruction segment comprises connecting the first reconstruction segment to which the inverse matrix has been applied and the second reconstruction segment based on the SDF value of the first reconstruction segment, an SDF value of the second reconstruction segment, and weight functions of the first reconstruction segment and the second reconstruction segment.

19. The camera tracking method of claim 7, wherein the camera is moved along a three-dimensional surface.

13. A camera tracking apparatus comprising:

a first graphic processing unit (GPU) to generate, using a processor, a first reconstruction segment from a first plurality of frames obtained by a camera, and store the generated first reconstruction segment; and

a second GPU to generate, when a preset reconstruction segment division condition is satisfied, using a processor, a second reconstruction segment from a second plurality of frames obtained by the camera, the second plurality of frames including a frame after the first reconstruction segment as a start frame, and store the generated second reconstruction segment,

wherein the second GPU generates a third reconstruction segment, transmits data of the second reconstruction segment to the first GPU when the third reconstruction segment is generated, and stores the generated third reconstruction segment,

wherein the first GPU connects the first reconstruction segment with the second reconstruction segment received from the second GPU, and

wherein the second GPU determines that the reconstruction division condition is satisfied when a movement vector of the camera is greater than a preset value.

14. The camera tracking apparatus of claim 13, further comprising a transformation matrix generation unit to generate, using a processor, a transformation matrix for the second reconstruction segment when the second reconstruction segment has a distortion, and to apply the generated transformation matrix to the second reconstruction segment transmitted from the second GPU to the first GPU to transform the second reconstruction segment.

15. The camera tracking apparatus of claim 14, wherein the transformation matrix generation unit generates a map set for the first reconstruction segment and the second reconstruction segment based on a virtual reference camera, the virtual reference camera being positioned between a first position of the camera corresponding to an end frame of the first reconstruction segment and a second position of the camera corresponding to a start frame of the second reconstruction segment, and

wherein the transformation matrix generation unit applies an Iterative Closest Point (ICP) algorithm to the generated map set to generate the transformation matrix.

16. The camera tracking apparatus of claim 15, wherein the transformation matrix generation unit applies an inverse matrix of the transformation matrix to a voxel grid domain of the second reconstruction segment.

17. The camera tracking apparatus of claim 16, wherein the first GPU connects the second reconstruction segment to which the inverse matrix has been applied and the first reconstruction segment based on a signed distance function (SDF) value of the second reconstruction segment, an SDF value of the first reconstruction segment, and weight functions of the first reconstruction segment and the second reconstruction segment.

20. The camera tracking apparatus of claim 13, wherein the camera is configured to move along a three-dimensional surface and to obtain the first and second pluralities of frames of the three-dimensional surface.