METHODS AND SYSTEMS FOR CONVERTING 2D MOTION PICTURES FOR STEREOSCOPIC 3D EXHIBITION

• US 20090116732A1
• Filed: 06/25/2007
• Published: 05/07/2009
• Est. Priority Date: 06/23/2006
• Status: Active Grant

• Current Assignee: IMAX Corporation
• Original Assignee: IMAX Corporation
• Inventors: Zhou, Samuel, Ye, Ping, Judkins, Paul
• Granted Patent: US 8,411,931 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion

Abstract

The present invention discloses methods of digitally converting 2D motion pictures or any other 2D image sequences to stereoscopic 3D image data for 3D exhibition. In one embodiment, various types of image data cues can be collected from 2D source images by various methods and then used for producing two distinct stereoscopic 3D views. Embodiments of the disclosed methods can be implemented within a highly efficient system comprising both software and computing hardware. The architectural model of some embodiments of the system is equally applicable to a wide range of conversion, re-mastering and visual enhancement applications for motion pictures and other image sequences, including converting a 2D motion picture or a 2D image sequence to 3D, re-mastering a motion picture or a video sequence to a different frame rate, enhancing the quality of a motion picture or other image sequences, or other conversions that facilitate further improvement in visual image quality within a projector to produce the enhanced images.

Claims

1. A method for processing at least one original image data sequence, the method comprising:

receiving original image data comprising at least one original image data sequence and processing information;

collecting a first set of processing information;

storing the first set of processing information in a first render data record;

generating a converted image data sequence based on the original image data sequence and first render data record;

verifying the converted image data sequence, wherein verifying the converted image data sequence comprises;

comparing the converted image data sequence to the original image data sequence to identify differences between the converted image data sequence and the original image data sequence; and

analyzing the identified differences to determine if updating or re-computing is needed; and

storing and outputting the first render data record if no updating or re-computing is needed.

2. The method of claim 1, wherein verifying the converted image data sequence further comprises:

determining processing information needs to be updated;

updating the processing information by collecting a second set of processing information;

updating the render data record by storing the second set of processing information in the render data record; and

re-generating the converted image data sequence based on the original image data sequence and the updated render data record.

3. The method of claim 1, wherein verifying the converted image data sequence further comprises:

determining the converted image data sequence is unacceptable;

updating the processing information by collecting a second set of processing information;

updating the render data record by storing the second set of processing information in the render data record;

re-generating the converted image data sequence based on the original image data sequence and the updated render data record; and

verifying the re-generated converted image data sequence to determine if updating or re-computing is needed; and

storing and outputting the updated render data record if no updating or re-computing is needed.

4. The method of claim 1, further comprising:

rendering the converted image data sequence for viewing and verification.

5. The method of claim 1, wherein generating the converted image data sequence based on the original image data sequence and first render data record further comprises:

generating the converted image data sequence having enhanced resolution and image quality.

6. The method of claim 1, wherein generating the converted image data sequence based on the original image data sequence and first render data record further comprises:

generating the converted image data sequence having a different frame rate than the original image data sequence.

7. A system to implement the method of claim 1, the system comprising:

a front-end system comprising an open platform adapted to;

accommodate commercial software providing render scripting capability; and

provide at least one of parallel and distributed computing; and

allow for human interaction; and

a backend system adapted to;

manage image sequence conversion processing;

perform automated functions; and

perform central database functions.

8. The system of claim 7, wherein the front-end system further comprises:

work station terminals having a graphical user interface, wherein the number of work station terminals is scalable;

a front-end system network switch adapted to interface with a backend system network switch;

a pre-render adapted to provide the parallel and distributed computing; and

a verification server adapted to output the verified converted image data sequence.

9. The system of claim 7, wherein the backend system further comprises:

an intelligent controller server adapted to manage image sequence conversion processing;

a central data storage;

a data input/output device adapted to receive the original image data sequence;

a backend system network switch adapted to interface with a front-end system network switch;

one or more render cluster processors adapted to perform automated processes, wherein the number of render cluster processors is scalable; and

a user controls interface adapted to allow users to monitor the image sequence conversion process.

10. The system of claim 8, wherein the front-end further comprises a display area for allowing a converted image sequence to be examined.

11. A method for determining a depth of an object in an image frame, the method comprising:

receiving camera modeling and image sequence data, the image sequence data comprising a plurality of image frames, each image frame comprising at least one object;

selecting at least one image frame from the image sequence data;

identifying each object in the selected image frame;

assigning an estimated dimension for each object;

creating a model based on the estimated dimension;

positioning the model at a first selected distance from a virtual camera position;

obtaining a virtual camera image of the model from the virtual camera view point;

positioning the model at a second selected distance from the virtual camera position and obtaining a second virtual camera image of the model until the second virtual camera image matches the estimated dimension of the object; and

determining an estimated depth of the object based on the first and second selected distances of the model to the virtual camera.

12. The method of claim 11, wherein an estimated dept of frame objects is performed for multiple image frames.

13. A method for converting a 2D image data sequence into a 3D image data sequence, the method comprising:

receiving the 2D image data sequence;

performing a temporal analysis on the 2D image data sequence to determine image motion cues;

dividing the 2D image data sequence into scenes, each scene comprising at least one object, wherein the scenes are grouped;

identifying objects in each scene and determining characteristics associated with each object, wherein the characteristics comprise a geometrical position and shape;

determining whether each scene is suitable for automated processing;

assigning each object to an object layer based on the characteristics associated with each object;

generating a depth map for each object based on the object layer;

collecting processing information associated with each object;

generating converted object images using the processing information;

storing the processing information into a render data record;

combining the converted object images to produce a 3D scene;

combining depth maps into full frame depth maps;

filling occlusion regions in the converted 3D scene;

generating a 3D image sequence using the full frame depth maps and 3D scene;

outputting the converted 3D image and render data record.

14. The method of claim 13, further comprising:

enhancing the 2D image data sequence before dividing the 2D image data sequence.

15. The method of claim 13, further comprising:

generating a left eye image data sequence and a right eye image data sequence using the 2D original image data sequence, wherein the left and right eye image data sequences form a stereoscopic 3D view.

16. The method of claim 15, where the original 2D image data sequence is used as the left or right image data sequence.

17. The method of claim 13, wherein the data render records comprise data associated with processing the 2D original image sequence and generating the 3D image sequence, the associated data comprising:

processing information;

editorial information;

computing instructions and parameters.

18. The method of claim 14, wherein performing a temporal analysis on the 2D image data sequence to determine image motion cues provides motion information to facilitate processing;

and

wherein enhancing the 2D image data sequence before dividing the 2D image data sequence provides enhancement information to facilitate processing in downstream stages.

19. The method of claim 13, wherein dividing the 2D image data sequence into scenes comprises creating a shot list with shot group information.

20. The method of claim 13, wherein identifying objects in each scene and determining characteristics associated with each object, further comprising:

characterizing the geometrical position using match by scaling.

21. The method of claim 13, wherein determining whether each scene is suitable for automated processing is based on at least one of motion field and depth of field.

22. The method of claim 13, wherein assigning each object to the object layer based on characteristics associated with each object further comprises:

assigning each object to different layers.

23. The method of claim 22, further comprising:

associating a label to each object, wherein the label identifies the assigned layer for the associated object.

24. The method of claim 23, wherein the label is a color selected from a predefined color palette.

25. The method of claim 22, wherein assigning each object to different layers is based on a criteria, the criteria comprising at least one of:

object transparent;

reconstruction of objects with occlusion;

automated object separation;

description of particle objects;

object size compensation; and

motion classification.

26. The method of claim 25, wherein object transparency comprises:

using transparency for at least one pixel in an object;

corresponding multiple pixel and depth values to different objects; and

wherein the different objects are assigned to different layers.

27. The method of claim 13, wherein collecting processing information associated with each object and generating converted object images using the processing information comprises:

object mask generation;

object depth modeling;

object re-construction;

object depth map generation;

object scaling; and

computing at least one stereoscopic image.

28. The method of claim 27, wherein object re-construction comprises:

detecting missing object information; and

reconstructing the missing object information in a layer independent of other layers.

29. The method of claim 27, wherein object scaling comprises:

compensating for miniaturization or reverse miniaturization.

30. The method of claim 27, wherein object mask generation comprises:

analyzing the 2D image data sequence, layer information, and motion cues associated with the 2D image data sequence;

selecting an operating mode, wherein the operating mode is selected from the group consisting of;

automated;

semi-automated; and

manual;

wherein each operating mode comprises at least one sub-process; and

generating an object mask using one or more sub-processes.

31. The method of claim 30, wherein the automated operating mode comprises a shape recovery sub-process.

32. The method of claim 30, wherein the semi-automated operating mode comprises:

color-based separation sub-process; and

graph-cut based separation sub-process.

33. The method of claim 30, wherein generating the object mask further comprises:

generating a mask at selected frames; and

interpolating the mask for all frames in a sequence based on motion tracking.

34. The method of claim 27, wherein object depth modeling comprises:

analyzing the 2D image data sequence, layer information, and motion cues associated with the 2D image data sequence;

selecting an operating mode, wherein the operating mode is selected from the group consisting of;

automated;

semi-automated; and

manual; and

generating a rough object depth map and rough object mask using at least one sub-process.

35. The method of claim 34, wherein the automated operating mode comprises:

an automated depth recovery sub-process; and

wherein the automated depth recovery sub-process comprises calculating a depth value by;

detecting camera motion; and

obtaining camera motion using the pixel motion and temporal filtering.

36. The method of claim 34, wherein the semi-automated operation mode comprises:

a bump-map depth modeling sub-process comprising;

defining a best fit approximate geometrical shape of the object; and

automatically adding object details based on the best fit, color and shading;

a feature-guided depth recovery sub-process comprising;

calculating a depth value by detecting camera motion and obtaining camera motion from pixel motion; and

wherein detecting camera motion is adapted to be at least partially controlled by a user; and

a mask-guided depth recovery sub-process comprising;

tracking an object mask to obtain object size and motion, wherein the tracking is adapted to be at least partially guided by a user; and

calculating depth values based on object size and motion.

37. The method of claim 34, wherein the manual operation mode comprises a solid depth modeling sub-process, wherein the model follows object motion and does not require full object detail or fit in the object boundary.

38. The method of claim 34, further comprising:

using a plurality of sub-processes to generate the rough object depth map and rough object mask;

scaling and combining the results of each sub-process to produce the object depth model.

39. The method of claim 34, further comprising:

refining the rough object depth mask to produce a depth map.

40. The method of claim 39, further comprising:

generating depth maps for at least two scenes; and

merging the depth maps for the at least two scenes by depth dissolving.

41. The method of claim 27, wherein object re-construction comprises:

detecting missing information in an object; and

filling missing information by;

tracking object motion to determine corresponding features in other frames;

registering the corresponding features; and

copying corresponding pixels from the other frames.

42. The method of claim 13, wherein combining depth maps into full frame depth maps further comprises:

combining all object depth maps from all layers to create a full frame depth map; and

combining 3D scenes into full frame 3D images with occlusion regions.

43. The method of claim 13, wherein filling occlusion regions in the converted 3D scene comprises:

analyzing 3D images with occlusion regions, the 2D image data sequence, and motion cues;

selecting an operating mode, wherein the operating mode is selected from the group consisting of;

automated;

semi-automated; and

manual;

wherein each operating mode comprises at least one sub-process;

filling occlusion regions using at least one sub-process; and

temporally filtering the filled occlusion regions to produce the converted 3D image sequence.

44. The method of claim 43, wherein the automated operating mode comprises a pixel recovery sub-process, the pixel recovery sub-process comprising:

tracking warped pixels in other frames; and

copying the tracked pixels.

45. The method of claim 43, wherein the semi-automated operating mode comprises:

a feature-guided pixel recovery sub-process comprising;

filling occlusion regions by copying tracked and warped pixels; and

wherein filling occlusion regions is adapted to receive user input;

a texture synthesis sub-process comprising;

synthesizing a texture of regions surrounding each region of occlusion; and

an in-painting sub-process;

propagating pixel information collected from outside occlusion regions over multiple frames.

46. The method of claim 43, further comprising:

filling occlusion regions using a plurality of sub-processes to produce a plurality of scenes having filled occlusion regions; and

selecting the scene using a voting process, wherein the voting process is based on temporal consistency and spatial smoothness.

47. A system to implement the method of claim 13, the system comprising:

a front-end system comprising an open platform adapted to;

accommodate commercial software providing render scripting capability; and

provide at least one of parallel and distributed computing; and

allow for human interaction; and

a backend system adapted to;

manage image sequence conversion processing;

perform automated functions; and

perform central database functions.

48. The system of claim 47, wherein the front-end system further comprises:

work station terminals having a graphical user interface, wherein the number of work station terminals is scalable;

a front-end system network switch adapted to interface with a backend system network switch;

a pre-render adapted to provide the parallel and distributed computing; and

a verification server adapted to output the verified converted image data sequence.

49. The system of claim 47, wherein the backend system further comprises:

an intelligent controller server adapted to manage image sequence conversion processing;

a central data storage;

a data input/output device adapted to receive the 2D image data sequence;

a backend system network switch adapted to interface with a front-end system network switch;

one or more render cluster processors adapted to perform automated processes, wherein the number of render cluster processors is scalable; and

a user controls interface adapted to allow users to monitor the image sequence conversion process.

50. The system of claim 48, wherein the front-end further comprises a display area for allowing a converted 3D image sequence to be examined.