Real-Time 3D Virtual or Physical Model Generating Apparatus for HoloPortal and HoloCloud System

• US 20170010584A1
• Filed: 07/09/2016
• Published: 01/12/2017
• Est. Priority Date: 07/09/2015
• Status: Active Grant

• Current Assignee: DoubleMe, Inc.
• Original Assignee: DoubleMe, Inc.
• Inventors: KIM, Heekwan
• Granted Patent: US 10,516,869 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G03H 1/0443 Digital holography, i.e. recording holograms with digital recording means holobject computation G03H1/0866
• CPC Class Codes: G03H 1/0476 Holographic printer G03H1/268 takes precedence
• CPC Class Codes: G03H 2001/0452 arranged to record an image of the object holographic microscope G03H2001/005
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G16B 25/00 ICT specially adapted for hybridisation; ICT specially adapted for gene or protein expression
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 21/23 Processing of content or additional data; Elementary server operations; Server middleware
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762

Abstract

A novel electronic system provides fast three-dimensional model generation, social content sharing of dynamic three-dimensional models, and monetization of the dynamic three-dimensional models created by casual consumers. In one embodiment, a casual consumer utilizes a dedicated real-time 3D model reconstruction studio with multiple camera angles, and then rapidly create dynamic 3D models with novel computational methods performed in scalable graphics processing units. In another embodiment, uncalibrated multiple sources of video recording of a targeted object are provided by a plurality of commonly-available consumer video recording devices (e.g. a smart phone, a camcorder, a digital camera, etc.) located at different angles, after which the uncalibrated multiple sources of video recording are transmitted to a novel cloud computing system for real-time temporal, spatial, and photometrical calibration and 3D model reconstruction. The dynamic 3D models can be uploaded, listed, and shared among content creators and viewers in an electronic sharing platform.

Claims

1. A real-time 3D virtual and physical body double-generating apparatus comprising:

a HoloPortal electronic system that incorporates a dedicated physical studio space, which is configured to capture, calibrate, calculate, reconstruct, and generate graphical transformation of a target object to create an electronic 3D body double model;

a HoloCloud electronic system comprising consumer-level portable video recording devices positioned circularly around the target object and generate a plurality of multiple-angle video data streams, a scalable number of graphics processing units (GPU'"'"'s) that execute a pre-processing module and a 3D reconstruction module for calibrating, calculating, reconstructing, and generating graphical transformations to create the electronic 3D body double model that resembles the target object, and a wireless transceiver that transmits the plurality of multiple-angle video data streams from the consumer-level portable video recording devices to the scalable number of GPU'"'"'s;

a 3D printer device connected to at least one of the HoloPortal electronic system and the HoloCloud electronic system, wherein the 3D printer device manufactures a physical replica of the electronic 3D body double model previously created by the HoloPortal electronic system or the HoloCloud electronic system, when a user requests production of the physical replica;

a 3D model and content database configured to store the electronic 3D body double model created from the HoloPortal electronic system or the HoloCloud electronic system;

an electronic 3D content sharing software executed on a computer server connected to the 3D model and content database, wherein the electronic 3D content sharing software is configured to upload, list, transmit, and share 3D model animations and 3D contents derived from the electronic 3D body double model, and wherein the electronic 3D content sharing software is further configured to accommodate a user request to purchase or sell the physical replica of the electronic 3D body double model; and

a client-side 3D content viewer and management user interface executed on a notebook computer, a desktop computer, a mobile communication device, or a web server, wherein the client-side 3D content viewer and management user interface is configured to purchase, sell, transmit, receive, or playback a 3D content incorporating the electronic 3D body double model via the electronic 3D content sharing software and the 3D model and content database.

2. The real-time 3D virtual and physical body double-generating apparatus of claim 1, further comprising a hologram display machine configured to project the electronic 3D body double model as a hologram after a user'"'"'s hologram projection request is processed via the client-side 3D content viewer and management user interface, the electronic 3D content sharing software, and the 3D model and content database.

3. The real-time 3D virtual and physical body double-generating apparatus of claim 1, wherein the pre-processing module calibrates temporal, spatial, and photometrical variables of the multiple-angle video data streams, and generates background 3D geometry and 360-degree virtual reality video data.

4. The real-time 3D virtual and physical body double-generating apparatus of claim 1, wherein the 3D reconstruction module provides depth map computations, voxel grid reconstructions, and deformed mesh generations for creation of the electronic 3D body double model, which resembles the target object.

5. The real-time 3D virtual and physical body double-generating apparatus of claim 1, wherein the client-side 3D content viewer and management user interface is a mobile application for the mobile communication device.

6. The real-time 3D virtual and physical body double-generating apparatus of claim 1, wherein the client-side 3D content viewer and management user interface is a computer application for the notebook computer or the desktop computer.

7. The real-time 3D virtual and physical body double-generating apparatus of claim 1, wherein the client-side 3D content viewer and management user interface is a web interface accessible via a web browser on a consumer'"'"'s electronic device, which is connected to the web server.

8. The real-time 3D virtual and physical body double-generating apparatus of claim 1, wherein the 3D model animations and 3D contents derived from the electronic 3D body double model are shared, bought, and sold from one entity to another entity over the electronic 3D content sharing software executed on the computer server connected to the 3D model and content database.