APPARATUS AND METHOD FOR CONVERGING REALITY AND VIRTUALITY IN A MOBILE ENVIRONMENT

• US 20120162372A1
• Filed: 12/21/2011
• Published: 06/28/2012
• Est. Priority Date: 12/22/2010
• Status: Abandoned Application

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: GHYME, Sang-Won
• Application Number:
  US13/333,459
• Publication Number:
  US 20120162372A1
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/156 Mixing image signals

Abstract

Disclosed herein are an apparatus and a method for converging reality and virtuality in a mobile environment. The apparatus includes an image processing unit, a real environment virtualization unit, and a reality and virtuality convergence unit. The image processing unit corrects real environment image data captured by at least one camera included in a mobile terminal. The real environment virtualization unit generates real object virtualization data virtualized by analyzing each real object of the corrected real environment image data in a three-dimensional (3D) fashion. The reality and virtuality convergence unit generates a convergent image, in which the real object virtualization data and at least one virtual object of previously stored virtual environment data are converged by associating the real object virtualization data with the virtual environment data, with reference to location and direction data of the mobile terminal.

Claims

1. An apparatus for converging reality and virtuality in a mobile environment, comprising:

an image processing unit for correcting real environment image data captured by at least one camera included in a mobile terminal;

a real environment virtualization unit for generating real object virtualization data virtualized by analyzing each real object of the corrected real environment image data in a three-dimensional (3D) fashion; and

a reality and virtuality convergence unit for generating a convergent image, in which the real object virtualization data and previously stored virtual environment data are converged by associating the real object virtualization data with the virtual environment data, with reference to location and direction data of the mobile terminal.

2. The apparatus as set forth in claim 1, wherein the real environment virtualization unit comprises:

a multi-image matching unit for generating disparity map data by analyzing the corrected real environment image data in a 3D fashion;

a 3D shape restoration unit for generating real object disparity map data for each individual real object using the disparity map data and generating partial 3D mesh data using the real object disparity map data; and

a mesh warping unit for generating completed 3D mesh data, by performing mesh warping that collects the partial 3D mesh data in various directions and joins and patches the partial 3D mesh data and then filling remaining empty mesh part.

3. The apparatus as set forth in claim 2, wherein the real environment virtualization unit further comprises an estimation conversion unit for generating estimated 3D mesh data by estimating an empty mesh part in the generated partial 3D mesh data with reference to the completed 3D mesh data and generating real object rigging data by using the estimated 3D mesh data

4. The apparatus as set forth in claim 3, wherein the estimation conversion unit generates a skeleton structure and motion data by using the estimated 3D mesh data, determines mesh deformation attributable to a motion of the real object by using the skeleton structure and the motion data, and generates the real object rigging data by using the mesh deformation.

5. The apparatus as set forth in claim 4, wherein the estimation conversion unit comprises a virtualization data generation unit for generating the real object virtualization data by using the completed 3D mesh data, the skeleton structure and the motion data, and the real object rigging data

6. The apparatus as set forth in claim 2, wherein the 3D shape restoration unit converts the real object disparity map data into point cloud data and then generates the partial 3D mesh data by using a triangulation method.

7. The apparatus as set forth in claim 2, wherein the virtualization data generation unit generates individual virtualized data for each individual real object using the completed 3D mesh data, the skeleton structure and the motion data, and the real object rigging data, and generates the real object virtualization data by collecting the individual virtualized data for each individual real object.

8. The apparatus as set forth in claim 1, wherein the reality and virtuality convergence unit generates convergent space data by converging the real object virtualization data and the virtual environment data with reference to the location and direction data of the mobile terminal, and generates the convergent image by rendering the convergent space data.

9. A method of converging reality and virtuality in a mobile environment, comprising:

correcting real environment image data captured by at least one camera included in a mobile terminal;

generating real object virtualization data virtualized by analyzing a real object of the corrected real environment image data in a 3D fashion;

receiving location and direction data of the mobile terminal; and

providing a convergent image by converging the real object virtualization data and previously stored virtual environment data to be converged, with reference to the location and direction data of the mobile terminal.

10. The reality and virtuality convergence method as set forth in claim 9, wherein the generating real object virtualization data comprises:

generating disparity map data by analyzing the corrected real environment image data in a 3D fashion;

generating real object disparity map data for each individual real object using the disparity map data and generating partial 3D mesh data by using the real object disparity map data; and

generating completed 3D mesh data, by performing mesh warping that collects the partial 3D mesh data in various directions and joins and patches the partial 3D mesh data and then filling remaining empty mesh part.

11. The reality and virtuality convergence method as set forth in claim 10, wherein the generating real object virtualization data comprises:

generating estimated 3D mesh data by estimating an empty mesh part in the generated partial 3D mesh data with reference to the completed 3D mesh data;

generating skeleton structure and motion data by analyzing the estimated 3D mesh data, and then analyzing a motion of the real object;

determining mesh deformation attributable to the motion of the real object and generating real object rigging data based on the determined mesh deformation; and

generating the real object virtualization data by using the completed 3D mesh data, the skeleton structure and motion data, and the real object rigging data.

12. The reality and virtuality convergence method as set forth in claim 9, wherein the providing a convergent image comprises:

generating convergent space data by converging the real object virtualization data and the virtual environment data with reference to the location and direction data of the mobile terminal; and

generating the convergent image by rendering the convergent space data.

13. The reality and virtuality convergence method as set forth in claim 10, wherein the generating partial 3D mesh data comprises:

converting the real object disparity map data into point cloud data; and

generating the partial 3D mesh data by applying a triangulation method to the point cloud data.