Systems and methods for digitally re-mastering or otherwise modifying motion pictures or other image sequences data

• US 20040130680A1
• Filed: 10/14/2003
• Published: 07/08/2004
• Est. Priority Date: 03/13/2002
• Status: Active Grant

• Current Assignee: IMAX Corporation
• Original Assignee: IMAX Corporation
• Inventors: Zhou, Samuel, Ye, Ping, Judkins, Paul
• Granted Patent: US 7,856,055 B2
• Time in Patent Office: Days
• US Class Current: 352/38
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• CPC Class Codes: H04N 7/0112 one of the standards corresponding to a cinematograph film standard
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

A process and methods of digital enhancement of motion pictures and other moving image sequences for the purpose of being exhibited in an alternative display format including a large format cinema are disclosed. The invention efficiently enhances image resolution and quality through a temporal filtering process and achieves high performance using automated or interactive statistical quality evaluation methods. A system specially designed for efficient temporal computing with a parallel and distributed computing configuration equipped with a variety of optimization schemes is also disclosed. The performance of the process and the system is optimized through an intelligent controller and is scalable to support any throughput requirements demanded for concurrent motion picture releases in the original format as well as in any alternative format.

Claims

1. A method for digitally re-mastering an image sequence with enhanced image resolution and quality, comprising:

converting the image sequence to a digital format that comprises original image data;

processing the original image data using temporal processing to produce enhanced image data having enhanced resolution and quality.

2. The method of claim 1 further comprising converting the enhanced image data to an alternative format.

3. The method of claim 1 further comprising reviewing the image sequence for appropriate resolution and quality.

4. The method of claim 1 wherein the image sequence is a motion picture originated in a 35 mm film format, and wherein converting the image sequence to a digital format comprises digitizing the film using a film scanner.

5. The method of claim 1 wherein the image sequence is a motion picture originated in a digital format, and wherein converting the image sequence to a digital format comprises data conversion from the original digital format.

6. The method of claim 1 wherein the image sequence is in any image sequence format containing images captured using a device with an optical or electronic sensor.

7. The method of claim 2 wherein the alternative format is a film format with enhanced resolution and quality, and wherein converting the enhanced image data to an alternative format comprises film recording using a film recorder.

8. The method of claim 2 wherein the alternative format is a digital display format with enhanced resolution and quality, and wherein converting the enhanced image data to an alternative format comprises data conversion to digital display format.

9. The method of claim 2 wherein the alternative format is a concurrent release of an original motion picture with enhanced resolution and quality.

10. The method of claim 1 wherein processing the original image data using temporal processing comprises a pre-processing process, a render process, a verification process, and a post-processing process.

11. The method of claim 10 wherein the pre-processing process comprises:

generating proxy image data;

receiving pre-processing decisions for the original image data on shot separation, re-framing, color correction, artifact identification and scene classification; and

using image processing tools based on the pre-processing decisions to prepare the original image data for the render process.

12. The method of claim 11 wherein the scene classification is performed by an automated algorithm based on scene analysis.

13. The method of claim 11 wherein the artifact identification and artifact removal is performed by an automated algorithm in which artifacts are identified by their unique characteristics both from a current frame and from neighboring frames and removed by predicted pixel values calculated based on inter-frame motion estimates or intra-frame interpolation.

14. The method of claim 11 wherein the artifact identification and artifact removal is performed by a semi-automated algorithm for interactively locating artifacts and automated removal using temporal processing.

15. The method of claim 10 wherein render process comprises:

receiving original image data separated into shots having image shot data, wherein each shot is a render job;

predicting a render parameter set for each render job;

distributing render jobs among multiple computing render clients;

processing the render jobs to produce render results;

assembling the distributed render results;

checking the integrity of the assembled render results;

evaluating temporal processing render quality of the assembled render results; and

modifying the render parameters for improving the render results and re-processing if necessary.

16. The method of claim 15 wherein predicting the render parameter set for a shot comprises:

locating key frames;

computing initial motion estimation settings;

computing motion estimates by using the initial motion estimation settings and applying motion estimation algorithms to the key frames; and

computing statistical temporal measures based on the computed motion estimates to generate the render parameter set.

17. The method of claim 16 wherein the statistical temporal measures include TSNR, Motion and FMD.

18. The method of claim 17 wherein the render parameter set is produced by generating a set of matching threshold values corresponding to a given FMD and further modified based on TSNR and Motion.

19. The method of claim 15 wherein processing the render jobs comprises:

temporal filtering;

image resizing; and

image sharpening.

20. The method of claim 19 wherein processing the render jobs further comprises:

lighting change compensation;

noise equalization;

computing statistical quality indicators; and

proxy generation.

21. The method of claim 15 wherein processing the render jobs comprises a temporal filtering method to provide enhanced image resolution and reduce noise, comprising:

computing motion estimates;

regulating motion fields; and

recovering details.

22. The method of claim 21 wherein the temporal filtering method further comprises using a multi-pass temporal filtering method;

a temporal pyramid filtering method;

or a serial temporal filtering method to reduce noise.

23. The method of claim 21 wherein lighting change compensation is used to improve the performance of the temporal filtering method.

24. The method of claim 21 wherein noise equalization is used to improve the performance of the temporal filtering method.

25. The method of claim 20 wherein the statistical quality indicators comprise PMR and ASR.

26. The method of claim 15 wherein the evaluation of temporal processing quality is based on PMR and ASR.

27. The method of claim 15 wherein the render parameters are modified based on PMR and ASR.

28. The method of claim 19 wherein the image sharpening is based on enhancement of multiple levels of image details.

29. The method of claim 15 wherein distributing render jobs among multiple render clients is performed by assigning an entire image shot to a single render client;

dividing an image shot into overlapping segments and distributing them to render clients;

or dividing an image frame into overlapping regions and distributing them to render clients.

30. The method of claim 15 wherein the verification process comprises:

visually checking a proxy version of the enhanced image data for completeness;

visually verifying image noise level, sharpness and detail preservation and modify render parameters for re-submission if necessary based on statistical measures;

visually verifying color, artifacts and motion comfort level;

issuing a pass stamp if a render image shot meets quality standard.

31. The method of claim 10 wherein the post-processing process comprises:

final artifact removal;

final color correction;

motion correction;

organizing images; and

output image data conversion.

32. The method of claim 31 wherein motion correction is performed by reducing motion strobing by adding motion blur and reducing extreme camera shaking by partial camera stabilization.

33. A system for digitally re-mastering an image sequence with enhanced image resolution and quality, comprising:

a central control computer server;

an image data file server configured as a client device of the central control server;

multiple computing render devices configured as client devices to the central control server for processing original image data using temporal processing to produce enhanced image data having enhanced resolution and quality;

multiple workstations for user-interactive operations relating to the processing of the original image data configured as client devices to the central control server;

a data storage device for storing image data and process data;

image data input and output devices; and

a computer network that provides communications between the central control server and all client devices.

34. The system of claim 33 where the computing render devices are standard computers configured in a parallel and distributed configuration designed for supporting temporal processing of multiple image sequences simultaneously.

35. The system of claim 33 wherein the central control computer server is capable of resource management;

quality optimization;

computing efficiency optimization;

production management;

system administration; and

user interactivity.

36. A method for digitally re-mastering an image sequence with enhanced image resolution and quality by a temporal filtering method, comprising:

computing motion estimates;

regulating motion fields; and

recovering details.

37. The method of claim 36 wherein the temporal filtering method further comprises using a multi-pass temporal filtering method;

a temporal pyramid filtering method;

or a serial temporal filtering method to reduce noise.

38. The method of claim 36 wherein lighting change compensation is used to improve the performance of the temporal filtering method.

39. The method of claim 36 wherein noise equalization is used to improve the performance of the temporal filtering method.

40. A method for digitally re-mastering an image sequence with enhanced image resolution and quality by using an image sharpening method based on enhancement of multiple levels of image details.