Method of converting 2D video to 3D video using 3D object models

• US 9,438,878 B2
• Filed: 09/17/2015
• Issued: 09/06/2016
• Est. Priority Date: 05/01/2013
• Status: Expired due to Fees

• Current Assignee: Legend3D, Inc.
• Original Assignee: Legend3D, Inc.
• Inventors: Niebla, Vicente Jr., Baldridge, Tony, Schad, Thomas, Jones, Scott
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US14/857,704
• Publication Number:
  US 20160005228A1
• Time in Patent Office: 355 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion

Abstract

Method for converting 2D video to 3D video using 3D object models. Embodiments of the invention obtain a 3D object model for one or more objects in a 2D video scene, such as a character. Object models may for example be derived from 3D scanner data; planes, polygons, or surfaces may be fit to this data to generate a 3D model. In each frame in which a modeled object appears, the location and orientation of the 3D model may be determined in the frame, and a depth map for the object may be generated from the model. 3D video may be generated using the depth map. Embodiments may use feature tracking to automatically determine object location and orientation. Embodiments may use rigged 3D models with degrees of freedom to model objects with parts that move relative to one another.

Claims

1. A method of converting 2D video to 3D video using 3D object models, comprising:

obtaining a 2D video;

defining a 3D space associated with a scene in said 2D video;

obtaining a 3D object model of an object viewable in said scene;

identifying one or more frames in said scene, each containing an image of at least a portion of said object;

defining a position and orientation of said 3D object model in said 3D space that aligns said 3D object model with the image of said at least a portion of said object in a frame of said one or more frames;

generating a depth map associated with the frame from said 3D object model positioned and oriented in said 3D space;

assigning depth values in said depth map to other elements in said scene;

generating a stereoscopic 3D image pair from the frame and from said depth map.

2. The method of claim 1, wherein said obtaining a 3D object model of said object comprises

obtaining 3D scanner data captured from said object;

and,

converting said 3D scanner data into said 3D object model.

3. The method of claim 2, wherein said obtaining said 3D scanner data comprises obtaining data from a time-of-flight system or a light-field system.

4. The method of claim 2, wherein said obtaining said 3D scanner data comprises obtaining data from a triangulation system.

5. The method of claim 2, wherein said converting said 3D scanner data into said 3D object model comprises

retopologizing said 3D scanner data to form said 3D object model from a reduced number of polygons or parameterized surfaces.

6. The method of claim 1 wherein said defining a position and orientation of said 3D object model in said 3D space comprises

selecting one or more features in said 3D object model, each having a coordinates in said 3D object model;

determining pixel locations of said one or more features in said one or more frames;

calculating said position and orientation of said 3D object model in said 3D space to align a projection of said coordinates in said 3D object model onto a camera plane with said pixel locations in a plurality of frames of said one or more frames.

7. The method of claim 6 wherein said determining pixel locations of said one or more features in said one or more frames comprises

selecting said pixel locations in one or more key frames;

tracking said features across one or more non-key frames using a computer.

8. The method of claim 1 further comprising

dividing said 3D object model into object parts, wherein said object parts may have motion relative to one another;

augmenting said 3D object model with one or more degrees of freedom that reflect said motion relative to one another of said object parts;

determining values of each of said one or more degrees of freedom that align said image of said at least a portion of said object in a plurality of frames of said one or more frames with said 3D object model modified by said values of said one or more degrees of freedom.

9. The method of claim 8 wherein said determining values of each of said one or more degrees of freedom comprises

selecting one or more features in each of said object parts, each having coordinates in said 3D object model;

determining pixel locations of said one or more features in said one or more frames;

calculating a position and orientation of one of said object parts and calculating said values of each of said one or more degrees of freedom to align a projection of said coordinates in said 3D model onto a camera plane with said pixel locations in the frame.

10. The method of claim 9 wherein said determining pixel locations of said one or more features in said one or more frames comprises

selecting said pixel locations in one or more key frames;

tracking said features across one or more non-key frames using a computer.

11. The method of claim 2, wherein said converting said 3D scanner data into said 3D object model comprises

obtaining at least one mask associated with at least one area within said 3D scanner data;

calculating a best fit for one or more planes using a computer based on said 3D scanner data associated with the at least one area associated with each of the at least one mask;

altering automatically using said computer, one or more of position, orientation, shape, depth or curve of said one or more planes in order to fit edges or corners of said one or more planes with another plane;

generating said 3D object model from said one or more planes.

12. The method of claim 11, wherein said generating said 3D object model from said one or more planes comprises

generating a high resolution 3D object model from said one or more planes;

retopologizing said high resolution 3D object model to form said 3D object model from a reduced number of polygons or parameterized surfaces.

13. The method of claim 11, wherein the obtaining at least one mask associated with at least one area within said 3D scanner data comprises automatically generating the at least one mask comprising the at least one area wherein the at least one area is over a predefined size and within a predefined depth range.

14. The method of claim 11, wherein the obtaining at least one mask associated with at least one area within said 3D scanner data comprises automatically generating the at least one mask comprising the at least one area wherein the at least one area comprises a boundary having a difference in luminance values over a predefined threshold.

15. The method of claim 11, wherein the calculating the best fit for one or more planes using a computer based on said 3D scanner data associated with the at least one area associated with each of the at least one mask comprises calculating a normal vector for the one Or more planes.

16. The method of claim 11, wherein the calculating the best fit for one or more planes using a computer based on said 3D scanner data associated with the at least one area associated with each of the at least one mask comprises calculating a regression fit for the one or more planes.

17. The method of claim 11, wherein the calculating the best fit for one or more planes using a computer based on said 3D scanner data associated with the at least one area associated with each of the at least one mask comprises calculating an orthogonal distance regression fit for the one or more planes.

18. The method of claim 11, wherein said one or more planes comprise a plurality of planes and wherein the method further comprises grouping two or more of said plurality of planes.

19. The method of claim 18, wherein each of said two or more planes are grouped with a link of a predefined minimum and maximum distance or with a link having a spring constant.

20. The method of claim 11, further comprising accepting an input to alter any combination of position, orientation, shape, depth or curve of said one or more planes.