Systems and methods for digitally re-mastering or otherwise modifying motion pictures or other image sequences data

• US 7,856,055 B2
• Filed: 03/13/2003
• Issued: 12/21/2010
• Est. Priority Date: 03/13/2002
• Status: Active Grant

• Current Assignee: IMAX Corporation
• Original Assignee: IMAX Corporation
• Inventors: Zhou, Samuel, Ye, Ping, Judkins, Paul
• Primary Examiner(s): CZEKAJ, DAVID J
• Application Number:
  US10/474,780
• Publication Number:
  US 20040130680A1
• Time in Patent Office: 2,840 Days
• Field of Search: 37524001-24007, 37524012-24016, 375/240.26
• US Class Current: 375/240.12
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• CPC Class Codes: H04N 7/0112 one of the standards corresponding to a cinematograph film standard
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

A process and methods of digital enhancement of motion pictures and other moving image sequences for the purpose of being exhibited in an alternative display format including a large format cinema are disclosed. The invention efficiently enhances image resolution and quality through a temporal filtering process and achieves high performance using automated or interactive statistical quality evaluation methods. A system specially designed for efficient temporal computing with a parallel and distributed computing configuration equipped with a variety of optimization schemes is also disclosed. The performance of the process and the system is optimized through an intelligent controller and is scalable to support any throughput requirements demanded for concurrent motion picture releases in the original format as well as in any alternative format.

Claims

1. A method for digitally enhancing image resolution and quality of image sequence data, comprising:

performing a pre-processing process comprising converting the image sequence to a digital format to produce converted image data that comprises original image data;

processing the converted image data to produce enhanced image data, wherein processing the converted image data comprises a render process comprising;

receiving converted image data separated into shots having image shot data, wherein each shot is a render job;

estimating initial motion estimation settings using at least part of the converted image data;

determining statistical measures based at least in part on the initial motion estimation settings, the statistical measures being associated with temporal characteristics of the converted image data;

predicting a render parameter set for each render job based at least in part on the statistical measures, the render parameter set comprising render parameters;

distributing render jobs among multiple computing render clients for processing the render jobs;

performing temporal image processing on the render jobs using the render parameters to produce render results;

assembling the render results into a continuous shot;

checking an integrity of the assembled render results to identify missing frames or incomplete frames and repairing the assembled render results by re-rendering the missing frames or the incomplete frames;

computing quality indicators based on the assembled render results;

evaluating processing render quality based on the quality indicators of the assembled render results;

modifying the render parameters to improve the render results and re-processing as determined by evaluation; and

determining the assembled render results are the enhanced image data if the quality indicators are within normal ranges;

performing a verification process and a post-processing process on the enhanced image data; and

outputting the enhanced image data.

2. The method of claim 1 further comprising converting the enhanced image data to an alternative format.

3. The method of claim 2 wherein the alternative format is a film format with enhanced resolution and quality, and wherein converting the enhanced image data to an alternative format comprises film recording using a film recorder.

4. The method of claim 2 wherein the alternative format is a digital display format with enhanced resolution and quality, and wherein converting the enhanced image data to an alternative format comprises data conversion to digital display format.

5. The method of claim 2 wherein the alternative format is a concurrent release of an original motion picture with enhanced resolution and quality.

6. The method of claim 1 further comprising receiving approval of the output of enhanced image data.

7. The method of claim 1 wherein the image sequence is a motion picture originated in a 35 mm film format, and wherein converting the image sequence to a digital format comprises digitizing the film using a film scanner.

8. The method of claim 1 wherein the image sequence is a motion picture originated in a digital format, and wherein converting the image sequence to a digital format comprises data conversion from the original digital format.

9. The method of claim 1 wherein the image sequence is in any image sequence format containing images captured using a device with an optical or electronic sensor.

10. The method of claim 1 wherein the pre-processing process comprises:

receiving at least one pre-processing decision from a user based on previewing the converted image data and previewing a proxy version of the converted image data, the at least one pre-processing decision relating to at least one of;

shot separation;

re-framing;

artifact identification;

scene classification;

cropping;

color correction;

image resizing; and

artifact removal; and

using image processing tools based on the at least one pre-processing decision to prepare the original image data for the render process.

11. The method of claim 10 wherein the scene classification is performed by an automated algorithm based on scene analysis.

12. The method of claim 10 wherein the artifact identification and artifact removal is performed by an automated algorithm in which artifacts are identified by their unique characteristics both from a current frame and from neighboring frames and removed by predicted pixel values calculated based on inter-frame motion estimates or intra-frame interpolation.

13. The method of claim 10 wherein the artifact identification and artifact removal is performed by a semi-automated algorithm for interactively locating artifacts and automated removal using temporal processing.

14. The method of claim 1 wherein predicting the render parameter set for a shot comprises:

locating key frames;

computing motion estimates by using the initial motion estimation settings and applying motion estimation algorithms to the key frames;

determining the statistical measures based at least in part on the initial motion estimation settings to generate the render parameter set; and

determining render parameters using the statistical measures and user preferences.

15. The method of claim 14 wherein the statistical measures comprises:

temporal signal-to-noise ratio;

Motion; and

fast matching distribution.

16. The method of claim 14, wherein estimating initial motion estimation settings comprises:

estimation of granularity of the image noise distribution;

estimation of matching region of support based on the estimated noise granularity;

estimation of global motion by calculating the average absolute motion between key frames; and

estimation of searching range based on estimated global motion.

17. The method of claim 1, wherein performing temporal image processing on the render jobs using the render parameters to produce render results comprises:

one or more temporal filtering schemes further comprising at least one of;

using a multi-pass temporal filtering method;

a temporal pyramid filtering method;

a serial temporal filtering method;

or

a single pass temporal filtering method;

image resizing; and

image sharpening based on modulation transfer function compensation.

18. The method of claim 17 wherein the temporal filtering schemes comprises:

computing motion estimates;

regulating motion fields; and

recovering details.

19. The method of claim 17 wherein the modulation transfer function compensation is based on selective enhancement of multiple levels of image details.

20. The method of claim 17, wherein processing of each of the distributed render jobs further comprises at least one of:

lighting change compensation and lighting change inverse transform;

noise equalization and inverse noise equalization;

or

proxy generation.

21. The method of claim 1 wherein the quality indicators comprise pixel matching ratio and absolute still ratio.

22. The method of claim 21, further comprising evaluating the performance of the temporal filtering is based on the quality indicators.

23. The method of claim 1 wherein evaluating processing render quality is based on pixel matching ratio and absolute still ratio.

24. The method of claim 1 wherein the render parameters are modified based on pixel matching ratio and absolute still ratio.

25. The method of claim 1 wherein distributing render jobs among multiple computing render clients is performed based on a load-balancing scheme by at least one of:

assigning an entire image shot to a single render client;

dividing an image shot into overlapping segments and distributing them to render clients; and

dividing an image frame into overlapping regions and distributing them to render clients.

26. The method of claim 1 wherein the verification process comprises:

receiving a pass stamp if a render image shot meets quality standards after;

visually checking a proxy version of the enhanced image data for completeness;

visually verifying image noise level, sharpness and detail preservation and modifying render parameters for re-submission if necessary based on statistical measures; and

visually verifying color, artifacts and motion comfort level.

27. The method of claim 1 wherein the post-processing process comprises at least one of:

final artifact removal;

final color correction;

motion correction;

organizing images; and

output image data conversion.

28. The method of claim 27 wherein motion correction comprises reducing motion strobing by adding motion blur and reducing extreme camera shaking by partial camera stabilization.

29. The method of claim 1, wherein temporal image processing comprises:

recovering image details from multiple image frames using a temporal filtering method with render parameter prediction;

evaluating the performance of the temporal filtering and modifying render parameters using quality indicators; and

sharpening image details by emphasizing a selected range of detail levels.

30. The method of claim 29, wherein the temporal filtering method further comprises at least one of:

a multi-pass temporal filtering method;

a temporal pyramid filtering method;

a serial temporal filtering method; and

a single pass temporal filtering method.

31. The method of claim 29, wherein the temporal filtering method further comprises:

a noise equalization method; and

a lighting change compensation method.

32. The method of claim 29, wherein sharpening image details comprises:

using varying kernel sizes to selectively emphasize image details at different detail levels; and

using a selective sharpening gain value at each detail level to compensate modulation transfer function degradation.

33. The method of claim 32, wherein using the selective sharpening gain value at each detail level to compensate modulation transfer function degradation comprises selectively enhancing multiple levels of image details.

34. The method of claim 29, wherein sharpening image details by emphasizing a selected range of detail levels comprises:

applying a first level of sharpening to a first level of detail of the converted image data; and

applying a second level of sharpening to a second level of detail of the converted image data.

35. The method of claim 34, wherein applying the first level of sharpening to the first level of detail of the converted image data comprises:

using a first kernel size to emphasize image details at the first level of detail; and

using a first sharpening gain value at the first level of detail to compensate modulation transfer function degradation,

wherein applying the second level of sharpening to the second level of detail of the converted image data comprises;

using a second kernel size to emphasize image details at the second level of detail; and

using a second sharpening gain value at the second level of detail to compensate modulation transfer function degradation.

36. The method of claim 34, wherein applying the first level of sharpening to the first level of detail of the converted image data and applying the second level of sharpening to the second level of detail of the converted image data is based, at least in part, on modulation transfer function compensation.

37. The method of claim 1, further comprising:

repeating the processing of converted image data if the quality indicators are not within the normal ranges.

38. The method of claim 1, wherein the enhanced image data comprises a number of pixels greater than the converted image data that comprises original image data.

39. The method of claim 1, wherein converting the image sequence, processing the converted image data, and outputting the enhanced image data are automatically controlled and allow user input and interaction.

40. The method of claim 1, wherein determining the assembled render results are the enhanced image data if the quality indicators are within the normal ranges comprises:

comparing the quality indicators to a standard profile comprising the normal ranges for the quality indicators.

41. A system for enhancing a resolution and quality of an image sequence, comprising:

an intelligent controller that automatically controls, monitors, and manages digitally enhancing the resolution and quality of an image sequence, the intelligent controller comprising;

a central control computer server; and

a controller data storage device for supporting the central control computer server;

a plurality of computing render devices for processing original image data to produce enhanced image data having enhanced resolution and quality, each render device of the render devices being automatically controlled and adapted to receive user input and interaction and configured as client devices to the central control server, wherein the multiple computing render devices are configured to produce enhanced image data by;

performing a pre-processing process by converting the image sequence to a digital format to produce converted image data that comprises original image data;

performing a render process by;

receiving converted image data separated into shots having image shot data, wherein each shot is a render job;

estimating initial motion estimation settings using at least part of the converted image data;

determining statistical measures based at least in part on the initial motion estimation settings, the statistical measures being associated with temporal characteristics of the converted image data;

predicting a render parameter set for each render job based at least in part on the statistical measures, the render parameter set comprising render parameters;

distributing render jobs among the plurality of computing render devices for processing the render jobs;

performing temporal image processing on the render jobs using the render parameters to produce render results;

assembling the render results into a continuous shot;

checking an integrity of the assembled render results to identify missing frames or incomplete frames and repairing the assembled render results by re-rendering the missing frames or the incomplete frames;

computing quality indicators based on the assembled render results;

evaluating processing render quality based on the quality indicators of the assembled render results;

modifying the render parameters to improve the render results and re-processing as determined by evaluation; and

determining the assembled render results are the enhanced image data if the quality indicators are within normal ranges; and

performing a verification process and a post-processing process on the enhanced image data.

42. The system of claim 41 where the computing render devices are standard computers configured in a parallel and distributed configuration designed for supporting temporal processing of multiple image sequences simultaneously.

43. The system of claim 41 wherein the central control computer server is capable of resource management;

quality optimization;

computing efficiency optimization;

production management;

system administration; and

user interactivity.

44. The system of claim 41, further comprising:

a data file server configured as a client device of the central control server;

multiple workstations for user input and interactive operations relating to the processing of the original image data and configured as client devices to the central control server;

a central data storage device for storing image data and process data;

image data input and output devices; and

a computer network that provides communications between the central control server and all client devices.

45. The system of claim 44, further comprising a central data storage comprising the controller data storage device and the central data storage device;

and

wherein the central control server comprises the data file server.

46. The system of claim 41, wherein the enhanced image data comprises a number of pixels greater than the converted image data that comprises original image data.

47. The system of claim 41, wherein the multiple computing render devices are capable of determining the assembled render results are the enhanced image data if the quality indicators are within the normal ranges by:

comparing the quality indicators to a standard profile comprising the normal ranges for the quality indicators.