Method of converting 2D video to 3D video using machine learning
First Claim
1. A machine learning method of converting 2D video to 3D video, comprising:
- obtaining a training set comprising a plurality of conversions, each conversion comprising a 2D scene comprising one or more 2D frames;
a corresponding 3D conversion dataset that describes conversion of said 2D scene to 3D, comprising inputs and outputs for 2D to 3D conversion steps, said 2D to 3D conversion steps comprising obtaining said one or more 2D frames;
locating and identifying an object in one or more object frames within said one or more 2D frames, each object frame containing an image of at least a portion of said object;
generating an object mask for said object in said one or more object frames, said object mask identifying one or more masked pixels representing said object in said one or more object frames;
generating an object depth model that assigns a pixel depth to one or more of said one or more masked pixels;
generating a stereoscopic image pair for each of said one or more object frames based on said object depth model, said stereoscopic image pair comprising a left image and a right image; and
,generating one or more gap filling pixel values for one or more missing pixels in said left image or in said right image;
training a machine learning system on said training set;
obtaining a 2D video;
applying said machine learning system to said 2D video to automatically perform one or more of said 2D to 3D conversion steps on said 2D video; and
,accepting input from an operator to modify or complete one or more of said 2D to 3D conversion steps on said 2D video.
6 Assignments
0 Petitions
Accused Products
Abstract
Machine learning method that learns to convert 2D video to 3D video from a set of training examples. Uses machine learning to perform any or all of the 2D to 3D conversion steps of identifying and locating objects, masking objects, modeling object depth, generating stereoscopic image pairs, and filling gaps created by pixel displacement for depth effects. Training examples comprise inputs and outputs for the conversion steps. The machine learning system generates transformation functions that generate the outputs from the inputs; these functions may then be used on new 2D videos to automate or semi-automate the conversion process. Operator input may be used to augment the results of the machine learning system. Illustrative representations for conversion data in the training examples include object tags to identify objects and locate their features, Bézier curves to mask object regions, and point clouds or geometric shapes to model object depth.
421 Citations
14 Claims
-
1. A machine learning method of converting 2D video to 3D video, comprising:
-
obtaining a training set comprising a plurality of conversions, each conversion comprising a 2D scene comprising one or more 2D frames; a corresponding 3D conversion dataset that describes conversion of said 2D scene to 3D, comprising inputs and outputs for 2D to 3D conversion steps, said 2D to 3D conversion steps comprising obtaining said one or more 2D frames; locating and identifying an object in one or more object frames within said one or more 2D frames, each object frame containing an image of at least a portion of said object; generating an object mask for said object in said one or more object frames, said object mask identifying one or more masked pixels representing said object in said one or more object frames; generating an object depth model that assigns a pixel depth to one or more of said one or more masked pixels; generating a stereoscopic image pair for each of said one or more object frames based on said object depth model, said stereoscopic image pair comprising a left image and a right image; and
,generating one or more gap filling pixel values for one or more missing pixels in said left image or in said right image; training a machine learning system on said training set; obtaining a 2D video; applying said machine learning system to said 2D video to automatically perform one or more of said 2D to 3D conversion steps on said 2D video; and
,accepting input from an operator to modify or complete one or more of said 2D to 3D conversion steps on said 2D video. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)
-
Specification