CAMARA TRACKING APPARATUS AND METHOD USING RECONSTRUCTION SEGMENTS AND VOLUMETRIC SURFACE

• US 20150139532A1
• Filed: 06/25/2014
• Published: 05/21/2015
• Est. Priority Date: 11/20/2013
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: KIM, Young Hee, KIM, Jin Ho, KIM, Ki Hong, LEE, Gil Haeng
• Granted Patent: US 9,489,740 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 7/579 from motion

Abstract

Provided are an apparatus and method for tracking a camera that reconstructs a real environment in three dimensions by using reconstruction segments and a volumetric surface. The camera tracking apparatus using reconstruction segments and a volumetric surface includes a reconstruction segment division unit configured to divide three-dimensional space reconstruction segments extracted from an image acquired by a camera, a transformation matrix generation unit configured to generate a transformation matrix for at least one reconstruction segment among the reconstruction segments obtained by the reconstruction segment division unit, and a reconstruction segment connection unit configured to rotate or move the at least one reconstruction segment according to the transformation matrix generated by the reconstruction segment division unit and connect the rotated and moved reconstruction segment with another reconstruction segment.

Claims

1. A camera tracking apparatus using reconstruction segments and a volumetric surface, the camera tracking apparatus comprising:

a reconstruction segment division unit configured to extract three-dimensional space reconstruction segments from an image acquired by a camera and divide the reconstruction segments;

a transformation matrix generation unit configured to generate a transformation matrix for a first reconstruction segment having a distortion among the reconstruction segments obtained by the reconstruction segment division unit; and

a reconstruction segment connection unit configured to rotate or move the first reconstruction segment having a distortion according to the transformation matrix generated by the transformation matrix generation unit, and connect the rotated or moved first reconstruction segment with a second reconstruction segment having no distortion.

2. The camera tracking apparatus of claim 1, wherein the reconstruction segment division unit divides the reconstruction segments at a timing in which a movement factor of the camera is greater than a preset value or in preset units of frame.

3. The camera tracking apparatus of claim 1, wherein the transformation matrix generation unit sets a reference camera at a point between a point of the camera corresponding to a start frame of the first reconstruction segment having a distortion and a point of the camera corresponding to an end frame of the second reconstruction segment having to distortion, and generates a map set for the first reconstruction segment and second reconstruction segment of the reference camera.

4. The camera tracking apparatus of claim 3, wherein the transformation matrix generation unit generates a map set including a vertex map and a normal map for the first reconstruction segment and the second reconstruction segment of the reference camera, and applies an Iterative Closest Point (ICP) algorithm to the generated map set to generate a transformation matrix of the first reconstruction segment.

5. The camera tracking apparatus of claim 4, wherein the reconstruction segment connection unit applies an inverse matrix of the transformation matrix generated by the transformation matrix generation unit to a voxel grid domain of the first reconstruction segment.

6. The camera tracking apparatus of claim 5, wherein the reconstruction segment connection unit applies the inverse matrix including a rotation matrix and a movement matrix to a singed distance function (SDF) value of the first reconstruction segment.

7. The camera tracking apparatus of claim 6, wherein the reconstruction segment connection unit connects the first reconstruction segment having the inverse matrix applied thereto and the second reconstruction segment on the basis of the SDF value of the first reconstruction segment, an SDF value of the second reconstruction segment, and weight functions of the first reconstruction segment and the second reconstruction segment.

8. A camera tracking method using reconstruction segments and a volumetric surface, the camera tracking method comprising:

extracting three-dimensional space reconstruction segments from an image acquired by a camera and dividing the reconstruction segments;

generating a transformation matrix for a first reconstruction segment having a distortion among the divided reconstruction segments;

converting the first reconstruction segment having a distortion according to the transformation matrix; and

connecting the converted first reconstruction segment with the second reconstruction segment having no distortion.

9. The camera tracking method of claim 8, wherein the dividing the reconstruction segments comprises dividing the reconstruction segments at a timing in which a movement of the camera is greater than a preset value or in preset units of frame.

10. The camera tracking method of claim 8, wherein the generating a transformation matrix comprises generating a map set for the first reconstruction segment and second reconstruction segment on the basis of a point of a reference camera, and the reference camera acquires an image at a point between a point of the camera corresponding to a start frame of the first reconstruction segment having a distortion and a point of the camera corresponding to an end frame of the second reconstruction segment having to distortion along a moving route of the camera.

11. The camera tracking method of claim 10, wherein the generating a transformation matrix comprises generating a map set including a vertex map and a normal map and applying an Iterative Closest Point (ICP) algorithm to the generated map set to generate the transformation matrix.

12. The camera tracking method of claim 11, wherein the converting the first reconstruction segment comprises applying an inverse matrix of the transformation matrix to a voxel grid domain of the first reconstruction segment.

13. The camera tracking method of claim 12, wherein the converting the first reconstruction segment comprises applying the inverse matrix including a rotation matrix and a movement matrix to a singed distance function (SDF) value of the first reconstruction segment.

14. The camera tracking method of claim 13, wherein the connecting the converted first reconstruction segment comprises connecting the first reconstruction segment having the inverse matrix applied thereto and the second reconstruction segment on the basis of the SDF value of the first reconstruction segment, an SDF value of the second reconstruction segment, and weight functions of the first reconstruction segment and the second reconstruction segment.

15. A camera tracking apparatus using reconstruction segments and a volumetric surface, the camera tracking apparatus comprising:

a first graphic processing unit (GPU) configured to generate a first reconstruction segment from an image acquired by a camera and store the generated first reconstruction segment; and

a second GPU configured to, when a preset reconstruction segment division condition is satisfied, generate a second reconstruction segment having a frame after the first reconstruction segment as a start frame and store the generated second reconstruction segment,

wherein the second GPU transmits data of the second reconstruction segment to the first GPU when a third reconstruction segment is generated and generates and stores the three reconstruction segment, and

the first GPU connects the first reconstruction segment with the second reconstruction segment received from the second GPU.

16. The camera tracking apparatus of claim 15, wherein the second GPU determines that the reconstruction division condition is satisfied when a movement vector of the camera is greater than a preset value or when a first reconstruction segment is generated in preset units of frame, and generates and stores the second reconstruction segment.

17. The camera tracking apparatus of claim 15, further comprising a transformation matrix generation unit configured to generate a transformation matrix for the second reconstruction segment when the second reconstruction segment has a distortion and applying the generated transformation matrix to the second reconstruction segment transmitted from the second GPU to the first GPU to transform the second reconstruction segment.

18. The camera tracking apparatus of claim 17, wherein the transformation matrix generation unit generates a map set for the first reconstruction segment and second reconstruction segment based on a virtual reference camera between a point of the camera corresponding to an end frame of the first reconstruction segment and a point of the camera corresponding to a start frame of the second reconstruction segment and applies an ICP algorithm to the generated map set to generate the transformation matrix.

19. The camera tracking apparatus of claim 18, wherein the transformation matrix generation unit applies an inverse matrix of the transformation matrix to a voxel grid domain of the second reconstruction segment.

20. The camera tracking apparatus of claim 19, wherein the first GPU connects the second reconstruction segment having the inverse matrix applied thereto and the first reconstruction segment on the basis of an SDF value of the second reconstruction segment, an SDF value of the first reconstruction segment, and weight functions of the first reconstruction segment and the second reconstruction segment.