METHODS AND SYSTEMS FOR DIGITALLY RE-MASTERING OF 2D AND 3D MOTION PICTURES FOR EXHIBITION WITH ENHANCED VISUAL QUALITY

• US 20100231593A1
• Filed: 01/29/2007
• Published: 09/16/2010
• Est. Priority Date: 01/27/2006
• Status: Active Grant

• Current Assignee: IMAX Corporation
• Original Assignee: IMAX Corporation
• Inventors: Ye, Ping, Judkins, Paul, Zhou, Samuel
• Granted Patent: US 8,842,730 B2
• Time in Patent Office: Days
• US Class Current: 345/428
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 19/52 by predictive encoding
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/537 Motion estimation other than block-based
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 5/253 Picture signal generating by scanning motion picture films or slide opaques, e.g. for telecine scanning details therefor H04N3/36 ; standard conversion therefor H04N7/0112
• CPC Class Codes: H04N 7/0112 one of the standards corresponding to a cinematograph film standard
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter

Abstract

The present invention relates to methods and systems for the exhibition of a motion picture with enhanced perceived resolution and visual quality. The enhancement of perceived resolution is achieved both spatially and temporally. Spatial resolution enhancement creates image details using both temporal-based methods and learning-based methods. Temporal resolution enhancement creates synthesized new image frames that enable a motion picture to be displayed at a higher frame rate. The digitally enhanced motion picture is to be exhibited using a projection system or a display device that supports a higher frame rate and/or a higher display resolution than what is required for the original motion picture.

Claims

1. A method for enhancing the quality of a motion picture image sequence, the method comprising:

receiving an original motion picture image sequence comprising digital data of a plurality of image frames;

applying a spatial resolution enhancement process and a temporal resolution enhancement process to the digital image sequence to create an enhanced image sequence;

wherein the enhancement processes are automatically controlled; and

wherein the enhanced image sequence has a greater frame rate than the original image sequence and the enhanced image sequence has greater image detail than the original image sequence.

2. The method of claim 1, further comprising:

synchronizing the enhanced image sequence to an audio track for the original image sequence.

3. The method of claim 1, wherein the original motion picture image sequence and the enhanced image sequence are two-dimensional (2D) sequences.

4. The method of claim 1, wherein the original motion picture image sequence and the enhanced image sequence are three-dimensional (3D) sequences.

5. The method of claim 1, wherein the original motion picture image sequence is in 3D and the enhanced image sequence is in 2D.

6. The method of claim 1, further comprising:

dividing the original motion picture image sequence into shots; and

performing the enhancement processes on each shot.

7. The method of claim 1, wherein the original motion picture image sequence is a single shot.

8. The method of claim 1, further comprising:

formatting the enhanced image sequence to a display presentation format; and

synchronizing the formatted enhanced image sequence to an audio track for the original image sequence.

9. The method of claim 1, wherein the spatial resolution enhancement process comprises:

a motion-based spatial resolution enhancement process; and

a learning-based resolution enhancement process.

10. The method of claim 9, further comprising:

applying the motion-based spatial resolution process to a 3D image sequence, wherein applying the motion-based spatial resolution process comprises;

disparity estimation;

display map regulation; and

detail discovery.

11. The method of claim 9, wherein the learning-based spatial resolution enhancement process comprises:

generating a codebook comprising codewords, each codeword being associated with a high-resolution pattern;

applying a clustering analysis to reduce the size of the codebook;

upsizing an original image of the original image sequence to a higher resolution, the original image comprising a plurality of pixels;

matching each pixel of the upsized image to a codeword; and

replacing each pixel by a central pixel of the high-resolution pattern associated with the matched codeword.

12. The method of claim 9, wherein the learning-based spatial resolution enhancement process comprises:

generating a codebook comprising codewords, each codeword being associated with a high-resolution pattern;

applying a clustering analysis to reduce the size of the codebook;

upsizing an original image of the original image sequence to a higher resolution, the original image comprising at least one block of pixels;

matching each block of pixels of the upsized image to a codeword;

replacing the block of pixels by a high-resolution pattern associated with the matched codeword using a transformation process to create an enhanced block of pixels;

applying a blending process to the enhanced block of pixels; and

applying a temporal filtering process.

13. The method of claim 1, wherein the temporal resolution enhancement process comprises:

pre-processing;

global motion estimation;

local motion estimation;

half-motion vector generation;

frame interpolation; and

artifact repair by temporal consistency check.

14. The method of claim 13, wherein pre-processing comprises:

generating an edge mask map from each image frame; and

generating a color segmentation map from each image frame.

15. The method of claim 13, wherein each image frame comprises a plurality of pixels, the global motion estimation comprising:

a. computing a gradient cross correlation matrix for each pixel;

b. calculating at least one feature point for each pixel based on the gradient cross correlation matrix;

c. matching at least one of the calculated feature point to a feature point of a next frame in the image sequence;

d. selecting at least four matched feature points;

e. estimating global motion based on the selected feature points; and

f. interactively repeating steps d and e by selecting different matched feature points until a global motion estimate is obtained for each pixel.

16. The method of claim 13, wherein the local motion estimation process comprises:

receiving motion estimates from the spatial resolution enhancement process;

using the motion estimates as initial motion estimates;

receiving edge mask maps and color segmentation maps from the pre-processing process;

using global motion estimates from the global motion estimation; and

computing local motion vectors based on a voting-based method.

17. The method of claim 16, wherein each image frame comprises a plurality of pixels;

and

wherein the voting-based method comprises;

applying at least one local motion estimation method to each pixel;

computing forward and backward motion vectors for each local motion estimation method; and

selecting a motion vector for each pixel from the forward and backward motion vectors using a voting process.

18. The method of claim 17, wherein local motion estimation method comprises at least one of:

block matching method;

feature propagation method;

modified optical flow method; and

integral block matching method.

19. The method of claim 16, wherein the voting-based method is a pyramid voting-based method.

20. The method of claim 17, wherein the voting process comprises at least one of:

selecting a motion vector by different edge values;

selecting a motion vector by coherence check;

selecting a motion vector by minimum matching errors;

selecting a motion vector through forward and backward checks;

selecting a motion vector using color segmentation maps.

21. The method of claim 13, wherein half-motion vector generation comprises:

determining a time interval of a synthesized image frame between a first image frame and a second image frame;

assigning half-forward motion vectors and half-backward motion vectors to each pixel of the synthesized frame; and

generating a half-accuracy mask corresponding to the synthesized frame, the half-accuracy mask marking a status of each pixel.

22. The method of claim 13, wherein frame interpolation comprises:

receiving half-motion vectors and half-accuracy masks generated from at least two image frames;

creating a synthesized frame based, at least in part, on the half-motion vectors and half-accuracy masks;

generating missing pixels of the synthesized frame by interpolation and averaging;

inserting the missing pixels into the synthesized frame; and

generating the enhanced image sequence having the synthesized image frames.

23. The method of claim 22, further comprising:

maintaining synchronization with an audio track of the original image sequence by adding at least one synthesized frame.

24. The method of claim 22, wherein the synthesized frame is created using pyramid representation of a pair of images.

25. The method of claim 22, wherein the synthesized frame is created using more than two images.

26. The method of claim 13, wherein artifact repair by temporal consistency check comprises:

identifying artifact pixels by half accuracy masks;

grouping artifact pixels by visibility;

checking temporal consistency based on grouping; and

automatically repairing temporal inconsistent artifact pixels.

27. The method of claim 13, further comprising a layered frame interpolation process.

28. The method of claim 1, further comprising removing artifacts by user interaction.

29. The method of claim 27, wherein removing artifacts comprises merging different versions of synthesized frames.

30. A system for enhancing the quality of a motion picture image sequence, the system comprising:

a back-end subsystem comprising;

a central data storage for storing an original motion picture image sequence and an enhanced image sequence, the original motion picture image sequence comprising digital data of a plurality of image frames;

a render client for configured to perform a spatial resolution enhancement process and a temporal resolution enhancement process on the original image sequence to create the enhanced image sequence; and

an intelligent controller for controlling the render client and accessing the central data storage; and

wherein the intelligent controller automatically controls the enhancement processes; and

wherein the enhanced image sequence has a greater frame rate than the original image sequence and the enhanced image sequence has greater detail than the original image sequence.

31. The system of claim 30, further comprising:

a front-end subsystem comprising a workstation for communicating with the intelligent controller, the workstation being adapted to provide user input and interaction in repairing artifacts in the enhanced image sequence, performing a quality control check of the enhanced image sequence, and segmenting the original image sequence.

32. The system of claim 31, wherein the workstation comprises multiple workstations.

33. The system of claim 32, wherein at least one of the multiple workstations comprises a quality control workstation.

34. The system of claim 30, wherein the original motion picture image sequence and the enhanced image sequence are in 2D.

35. The system of claim 30, wherein the original motion picture image sequence and the enhanced image sequence are in 3D.

36. The system of claim 30, wherein the original motion picture image sequence is in 3D and the enhanced imaged sequence is in 2D.

37. The system of claim 30, wherein the render client is adapted to examine the quality of the enhanced image sequence.

38. The system of claim 30, wherein the intelligent controller comprises a processor and a memory comprising executable code, the executable code comprising:

a scheduler;

a auto launch;

a file setup; and

a tape writer.

39. The system of claim 30, wherein the render client comprises multiple render clients.

40. The system of claim 39, wherein the intelligent controller is adapted to detect a system failure and shut down each render client and re-assign a job to each render client.

41. The system of claim 39, wherein the intelligent controller is adapted to monitor the render clients to prevent re-rendering of data.

42. A method for enhancing the quality of an original motion picture image sequence, the method comprising:

receiving an image sequence;

receiving initial motion estimates;

applying a temporal resolution enhancement process comprising;

pre-processing;

global motion estimation;

local motion estimation;

half-motion vector generation;

frame interpolation; and

artifact repair by temporal consistency check;

wherein the temporal resolution enhancement process is automatically controlled; and

wherein the enhanced image sequence has a greater frame rate than the original image sequence.

43. The method of claim 42, wherein the original image sequence and enhanced image sequence is digital data.

44. The method of claim 42, wherein the original image sequence and the enhanced image sequence are in 2D.

45. The method of claim 42, wherein the original image sequence and the enhanced image sequence are in 3D.

46. The method of claim 42, wherein the original image sequence is in 3D and the enhanced imaged sequence is in 2D.

47. A method for enhancing the quality of an original motion picture image sequence, the method comprising:

receiving a 3D original motion picture image sequence;

applying a spatial resolution enhancement process to the 3D original image sequence to create an enhanced image sequence; and

wherein the enhanced image sequence has greater image detail than the original image sequence.

48. The method of claim 47, wherein the spatial resolution enhancement process comprises

a motion-based spatial resolution enhancement process;

and

a learning-based resolution enhancement process.

49. The method of claim 48, further comprising:

applying the motion-based spatial resolution process to a 3D image sequence, wherein applying the motion-based spatial resolution process comprises;

disparity estimation;

display map regulation; and

detail discovery.

50. The method of claim 48, wherein the learning-based spatial resolution enhancement process comprises:

generating a codebook comprising codewords, each codeword being associated with a high-resolution pattern;

applying a clustering analysis to reduce the size of the codebook;

upsizing an original image of the original image sequence to a higher resolution, the original image comprising a plurality of pixels;

matching each pixel of the upsized image to a codeword; and

replacing each pixel by a central pixel of the high-resolution pattern associated with the matched codeword.

51. The method of claim 48, wherein the learning-based spatial resolution enhancement process comprises:

generating a codebook comprising codewords, each codeword being associated with a high-resolution pattern;

applying a clustering analysis to reduce the size of the codebook;

upsizing an original image of the original image sequence to a higher resolution, the original image comprising at least one block of pixels;

matching each block of pixels of the upsized image to a codeword;

replacing the block of pixels by a high-resolution pattern associated with the matched codeword using a transformation process to create an enhanced block of pixels;

applying a blending process to the enhanced block of pixels; and

applying a temporal filtering process.