Methods and systems for converting 2D motion pictures for stereoscopic 3D exhibition

• US 8,411,931 B2
• Filed: 06/25/2007
• Issued: 04/02/2013
• Est. Priority Date: 06/23/2006
• Status: Active Grant

• Current Assignee: IMAX Corporation
• Original Assignee: IMAX Corporation
• Inventors: Zhou, Samuel, Judkins, Paul, Ye, Ping
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US12/299,909
• Publication Number:
  US 20090116732A1
• Time in Patent Office: 2,108 Days
• Field of Search: 382/100, 382/107, 382/154, 382/232, 382/236, 382/103, 382168-172, 348/42, 348/45, 348/46, 348/47, 348/48, 348/65, 348/66, 348/79, 348/80, 375/240.21, 375/240.29, 375/240.16, 375/240.02, 375/240.25, 375/240.26
• US Class Current: 382/154
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion

Abstract

The present invention discloses methods of digitally converting 2D motion pictures or any other 2D image sequences to stereoscopic 3D image data for 3D exhibition. In one embodiment, various types of image data cues can be collected from 2D source images by various methods and then used for producing two distinct stereoscopic 3D views. Embodiments of the disclosed methods can be implemented within a highly efficient system comprising both software and computing hardware. The architectural model of some embodiments of the system is equally applicable to a wide range of conversion, re-mastering and visual enhancement applications for motion pictures and other image sequences, including converting a 2D motion picture or a 2D image sequence to 3D, re-mastering a motion picture or a video sequence to a different frame rate, enhancing the quality of a motion picture or other image sequences, or other conversions that facilitate further improvement in visual image quality within a projector to produce the enhanced images.

Claims

1. A method for converting a 2D image data sequence into a 3D image data sequence, the method comprising:

receiving the 2D image data sequence;

performing a temporal analysis on the 2D image data sequence to determine image motion cues;

dividing the 2D image data sequence into scenes, each scene comprising at least one object, wherein the scenes are grouped;

identifying objects in each scene and determining characteristics associated with each object, wherein the characteristics comprise a geometrical position and shape;

determining whether each scene is suitable for automated processing based on scene classification;

assigning each object to an object layer based on the characteristics associated with each object;

generating a depth map for each object based on the object layer;

collecting processing information associated with each object;

generating converted object images using the processing information;

storing the processing information into a render data record;

combining the converted object images to produce a 3D scene;

combining depth maps into full frame depth maps;

filling occlusion regions in the converted 3D scene;

generating a 3D image sequence using the full frame depth maps and 3D scene;

outputting the converted 3D image and render data record.

2. The method of claim 1, wherein generating the depth map for each object based on the object layer comprises:

data;

implementing a virtual camera by receiving camera parameters;

identifying an object in a selected image frame;

receiving an estimated dimension for the object;

creating a model of the object based on the estimated dimension;

positioning the model of the object at a first selected distance from a virtual camera position;

computing an image of the model of the object from a virtual camera view point;

comparing a size of the model of the object in the image to a size of the object in the selected image frame;

positioning the model of the object at a second selected distance from the virtual camera position and computing a second image of the model of the object until the size of the model of the object in the second image matches the size of the object in the selected image frame; and

estimating a real-world position of the object based on the first and second selected distances of the model of the object for computing the depth of the object.

3. The method of claim 2, wherein estimating real-world positions of objects is performed for multiple image frames.

4. The method of claim 1, further comprising:

enhancing the 2D image data sequence before dividing the 2D image data sequence.

5. The method of claim 1, further comprising:

generating a left eye image data sequence and a right eye image data sequence using the 2D original image data sequence, wherein the left and right eye image data sequences form a stereoscopic 3D view.

6. The method of claim 5, where the original 2D image data sequence is used as the left or right image data sequence.

7. The method of claim 1, wherein the data render records comprise data associated with processing the 2D original image sequence and generating the 3D image sequence, the associated data comprising:

processing information;

editorial information;

computing instructions and parameters.

8. The method of claim 4, wherein performing a temporal analysis on the 2D image data sequence to determine image motion cues provides motion information to facilitate processing;

and

wherein enhancing the 2D image data sequence before dividing the 2D image data sequence provides enhancement information to facilitate processing in downstream stages.

9. The method of claim 1, wherein dividing the 2D image data sequence into scenes comprises creating a shot list with shot group information.

10. The method of claim 1, wherein identifying objects in each scene and determining characteristics associated with each object, further comprising:

characterizing the geometrical position using match by scaling.

11. The method of claim 1, wherein determining whether each scene is suitable for automated processing is based on at least one of motion field and depth of field.

12. The method of claim 1, wherein assigning each object to the object layer based on characteristics associated with each object further comprises:

assigning each object to different layers.

13. The method of claim 12, further comprising:

associating a label to each object, wherein the label identifies the assigned layer for the associated object.

14. The method of claim 13, wherein the label is a color selected from a predefined color palette.

15. The method of claim 12, wherein assigning each object to different layers is based on a criteria, the criteria comprising at least one of:

object transparent;

reconstruction of objects with occlusion;

automated object separation;

description of particle objects;

object size compensation; and

motion classification.

16. The method of claim 15, wherein object transparency comprises:

using transparency for at least one pixel in an object;

corresponding multiple pixel and depth values to different objects; and

wherein the different objects are assigned to different layers.

17. The method of claim 1, wherein collecting processing information associated with each object and generating converted object images using the processing information comprises:

object mask generation;

object depth modeling;

object re-construction;

object depth map generation;

object scaling; and

computing at least one stereoscopic image.

18. The method of claim 17, wherein object re-construction comprises:

detecting missing object information; and

reconstructing the missing object information in a layer independent of other layers.

19. The method of claim 17, wherein object scaling comprises:

compensating for miniaturization or reverse miniaturization.

20. The method of claim 17, wherein object mask generation comprises:

analyzing the 2D image data sequence, layer information, and motion cues associated with the 2D image data sequence;

selecting an operating mode, wherein the operating mode is selected from the group consisting of;

automated;

semi-automated; and

manual;

wherein each operating mode comprises at least one sub-process; and

generating an object mask using one or more sub-processes.

21. The method of claim 20, wherein the automated operating mode comprises a shape recovery sub-process.

22. The method of claim 20, wherein the semi-automated operating mode comprises:

color-based separation sub-process; and

graph-cut based separation sub-process.

23. The method of claim 20, wherein generating the object mask further comprises:

generating a mask at selected frames; and

interpolating the mask for all frames in a sequence based on motion tracking.

24. The method of claim 17, wherein object depth modeling comprises:

analyzing the 2D image data sequence, layer information, and motion cues associated with the 2D image data sequence;

selecting an operating mode, wherein the operating mode is selected from the group consisting of;

automated;

semi-automated; and

manual; and

generating a rough object depth map and rough object mask using at least one sub-process.

25. The method of claim 24, wherein the automated operating mode comprises:

an automated depth recovery sub-process; and

wherein the automated depth recovery sub-process comprises calculating a depth value by;

detecting camera motion; and

obtaining camera motion using the pixel motion and temporal filtering.

26. The method of claim 24, wherein the semi-automated operation mode comprises:

a bump-map depth modeling sub-process comprising;

defining a best fit approximate geometrical shape of the object; and

automatically adding object details based on the best fit, color and shading;

a feature-guided depth recovery sub-process comprising;

calculating a depth value by detecting camera motion and obtaining camera motion from pixel motion; and

wherein detecting camera motion is adapted to be at least partially controlled by a user; and

a mask-guided depth recovery sub-process comprising;

tracking an object mask to obtain object size and motion, wherein the tracking is adapted to be at least partially guided by a user; and

calculating depth values based on object size and motion.

27. The method of claim 24, wherein the manual operation mode comprises a solid depth modeling sub-process, wherein the model follows object motion and does not require full object detail or fit in the object boundary.

28. The method of claim 24, further comprising:

using a plurality of sub-processes to generate the rough object depth map and rough object mask;

scaling and combining the results of each sub-process to produce the object depth model.

29. The method of claim 24, further comprising:

refining the rough object depth mask to produce a depth map.

30. The method of claim 29, further comprising:

generating depth maps for at least two scenes; and

merging the depth maps for the at least two scenes by depth dissolving.

31. The method of claim 17, wherein object re-construction comprises:

detecting missing information in an object; and

filling missing information by;

tracking object motion to determine corresponding features in other frames;

registering the corresponding features; and

copying corresponding pixels from the other frames.

32. The method of claim 1, wherein combining depth maps into full frame depth maps further comprises:

combining all object depth maps from all layers to create a full frame depth map; and

combining 3D scenes into full frame 3D images with occlusion regions.

33. The method of claim 1, wherein filling occlusion regions in the converted 3D scene comprises:

analyzing 3D images with occlusion regions, the 2D image data sequence, and motion cues;

selecting an operating mode, wherein the operating mode is selected from the group consisting of;

automated;

semi-automated; and

manual;

wherein each operating mode comprises at least one sub-process;

filling occlusion regions using at least one sub-process; and

temporally filtering the filled occlusion regions to produce the converted 3D image sequence.

34. The method of claim 33, wherein the automated operating mode comprises a pixel recovery sub-process, the pixel recovery sub-process comprising:

tracking warped pixels in other frames; and

copying the tracked pixels.

35. The method of claim 33, wherein the semi-automated operating mode comprises:

a feature-guided pixel recovery sub-process comprising;

filling occlusion regions by copying tracked and warped pixels; and

wherein filling occlusion regions is adapted to receive user input;

a texture synthesis sub-process comprising;

synthesizing a texture of regions surrounding each region of occlusion; and

an in-painting sub-process;

propagating pixel information collected from outside occlusion regions over multiple frames.

36. The method of claim 33, further comprising:

filling occlusion regions using a plurality of sub-processes to produce a plurality of scenes having filled occlusion regions; and

selecting the scene using a voting process, wherein the voting process is based on temporal consistency and spatial smoothness.

37. A system, comprising:

a front-end system comprising an open platform adapted to;

accommodate commercial software providing render scripting capability;

provide at least one of parallel and distributed computing; and

allow for human interaction; and

a backend system adapted to;

manage converting a 2D image data sequence into a 3D image data sequence by;

receiving the 2D image data sequence;

performing a temporal analysis on the 2D image data sequence to determine image motion cues;

dividing the 2D image data sequence into scenes, each scene comprising at least one object, wherein the scenes are grouped;

identifying objects in each scene and determining characteristics associated with each object, wherein the characteristics comprise a geometrical position and shape;

determining whether each scene is suitable for automated processing based on scene classification;

assigning each object to an object layer based on the characteristics associated with each object;

generating a depth map for each object based on the object layer;

collecting processing information associated with each object;

generating converted object images using the processing information;

storing the processing information into a render data record;

combining the converted object images to produce a 3D scene;

combining depth maps into full frame depth maps;

filling occlusion regions in the converted 3D scene;

generating a 3D image sequence using the full frame depth maps and 3D scene;

outputting the converted 3D image and render data record;

perform automated functions; and

perform central database functions.

38. The system of claim 37, wherein the front-end system further comprises:

work station terminals having a graphical user interface, wherein the number of work station terminals is scalable;

a front-end system network switch adapted to interface with a backend system network switch;

a pre-render adapted to provide the parallel and distributed computing; and

a verification server adapted to output the verified converted image data sequence.

39. The system of claim 38, wherein the front-end further comprises a display area for allowing a converted 3D image sequence to be examined.

40. The system of claim 37, wherein the backend system further comprises:

an intelligent controller server adapted to manage image sequence conversion processing;

a central data storage;

a data input/output device adapted to receive the 2D image data sequence;

a backend system network switch adapted to interface with a front-end system network switch;

one or more render cluster processors adapted to perform automated processes, wherein the number of render cluster processors is scalable; and

a user controls interface adapted to allow users to monitor the image sequence conversion process.