System and method for automatically processing image data to provide images of optimal perceptual quality

• US 5,694,484 A
• Filed: 05/15/1995
• Issued: 12/02/1997
• Est. Priority Date: 05/15/1995
• Status: Expired due to Term

• Current Assignee: Intellectual Ventures I LLC (Intellectual Ventures LLC)
• Original Assignee: Polaroid Corporation (PLR IP Holdings LLC)
• Inventors: Cottrell, F. Richard, Hultgren, Bror O. III
• Primary Examiner(s): Couso, Jose L.
• Application Number:
  US08/440,647
• Time in Patent Office: 932 Days
• Field of Search: 382/100, 382/112-113, 382/128, 382/130-132, 382/141, 382/254-255, 382/307, 382/309, 382/311, 382/232, 358/448-467, 358/452-453, 358/400, 358/474, 358/487, 358/504, 358/508, 358/518, 358/520-522, 358/527, 358/530, 358/534, 358/135, 358/136
• US Class Current: 382/167
• CPC Class Codes: G06T 2207/20052 Discrete cosine transform [DCT]
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening

Abstract

An image processing system that automatically optimizes the perceptual quality of images undergoing a series of selected image-processing operations. The automatic optimization capability allows the system to bypass expensive and time-consuming trial-and-error methods associated with interactive methods. The system consists of a set of image-processing operations, an architecture, and an intelligent control. These elements take into consideration profiles of sources from which the images are generated, profiles of intended applications, and the impact that image processing operations (individually or in concert) will have on perceived image quality. The analysis uses a body of relationships linking human perception of image quality with objective metrics (such as sharpness, grain, tone, and color) of image content. The relationships used are based upon extensive psychovisual testing, using human observers and photographic images. The intelligent control embodies the test results and thereby functions as a synthetic human observer. Controlling a collection of adjustable parameters in the requested image-processing operations, it works automatically to maximize subjective quality for the resulting picture. When optimal parameter values have been determined, the system then subjects the image data, itself, to the corresponding processing operations and provides an image to the end application. This is all achieved without operator intervention other than initially selecting the set of processing operations. The functionality of the system may be implemented in a variety of physical architectures, including computer workstations, and including communicating sets of discrete components in which the computations and control are distributed among the components.

Claims

1. An image processing system for receiving image data from an image data source, performing at least one image processing operation of a plurality image processing operations in connection with image processing selection information to generate psychovisually optimized processed image data, and providing the psychovisually optimized processed image data to a downstream utilization element, the image processing system characterized by:

A. an image data processing section for performing said at least one image processing operation in connection with image data input thereto, the image data processing section performing said at least one image processing operation in relation to processing operation parameter information to generate said psychovisually optimized processed image data; and

B. a characteristic processing section for receiving source characteristic information and downstream utilization element characteristic information defining at least one selected characteristic of said image data source and downstream utilization element and for generating said processing operation parameter information in response to the image processing selection information, the source characteristic information, the downstream utilization element characteristic information, and predetermined psychovisual attributes, said processing operation parameter information enabling the image data processing section to generate the psychovisually optimized processed image data.

2. The image processing system of claim 1 in which the image data processing section includes a plurality of image processing atoms each for performing one of said plurality of image processing operations, and an image processing control for selecting ones of said image processing atoms and enabling the selected image processing atoms to process the image data in response to the image processing selection information.

3. The image processing system of claim 2 in which one of said plurality of image processing atoms operates in response to the processing operation parameter information generated by the characteristic processing section, the image processing control enabling said one of said plurality of image processing operations in response to the processing operation parameter information generated in response to the image processing selection information.

4. The image processing system of claim 2 in which at least two of said plurality of image processing atoms comprise an image processing molecule for performing ones of said plurality of image processing operations in a predefined order, the image processing control selecting and enabling each said image processing molecule and atoms therein in response to the image processing selection information.

5. The image processing system of claim 4 in which at least two of the image processing molecules comprise a molecule class, said at least two of the image processing molecules in each said molecule class performing a preselected type of image processing operation, the image processing control selecting and enabling each said image processing molecule and atoms therein in response to the image processing selection information.

6. The image processing system of claim 5 in which one said molecule class is a data decompression molecule class including a plurality of image processing molecules each for performing image data decompression in accordance with a selected decompression methodology, the image processing control selecting one of said plurality of image processing molecules from the decompression molecule class in response to decompression methodology information.

7. The image processing system of claim 5 in which one said molecule class is an image integrity molecule class including a plurality of image integrity molecules each for performing a series of image processing operations in connection with spatial image information, the image processing control selecting one of said plurality of image integrity molecules from the image integrity molecule class in response to image integrity selection information in said image processing selection information.

8. The image processing system of claim 5 in which one molecule class is a special effects molecule class including a plurality of special effects molecules each for performing a series of special effects image processing operations, the image processing control selecting one of said plurality special effects molecules from the special effects molecule class in response to special effects selection information in said image processing selection information.

9. The image processing system of claim 5 in which one molecule class is a compression molecule class including a plurality of image processing molecules each for performing image data compression in accordance with a selected compression methodology, the image processing control selecting one of said plurality of image processing molecules from the compression molecule class in response to compression methodology information in said image processing selection information.

10. The image processing system of claim 1 in which said characteristic processing section includes:

A. a parameter optimization element for generating trial parameter values;

B. a reference image quality value determination element for generating, in response to a reference image and said trial parameter values, a reference image quality value from both the source characteristic information and the downstream utilization element characteristic information;

C. an iteration control element for controlling the parameter optimization element and said reference image quality value determination element to enable the parameter optimization element through a series of iterations to generate said trial parameter values for providing said reference image quality value having an optimal value, and for providing said image data processing section with said processing operation parameter information, being said trial parameter values, when said reference image quality value equals said optimal value.

11. The image processing system of claim 10 in which the reference image quality value determination element comprises:

A. a reference profile generator for performing a processing operation corresponding to said at least one image processing operation performed by the image data processing section in connection with the source characteristic information and the downstream utilization element characteristic information to generate a plurality of reference image profile values; and

B. a quality value determination element for generating said reference image quality value in response to the plurality of reference image profile values.

12. The image processing system of claim 11 in which the source characteristic information and the downstream utilization element characteristic information both include modulation transfer function information and noise power spectrum information, said reference profile generator generating both a modulation transfer function profile value in response to the modulation transfer function information and a noise power spectrum profile value in response to the noise power spectrum information, said quality value determination element further generating said reference image quality value in response to both the modulation transfer function profile value and the noise power spectrum profile value.

13. The image processing system of claim 12 in which the quality value determination element further generates a sharpness quality value in response to the modulation transfer function profile value and a spectral weighting factor, said quality value determination element further generating said reference image quality value in response to the sharpness quality value.

14. The image processing system of claim 12 in which the quality value determination element further generates a graininess value in response to the noise power spectrum profile value and a spectral weighting factor, said quality value determination element further generating said reference image quality value in response to the graininess value.

15. An image processing method for processing image data received from an image data source, performing at least one image processing operation in connection with image processing selection information to generate psychovisually optimized processed image data, and providing the psychovisually optimized processed image data to a downstream utilization element, said method characterized by:

A. receiving source characteristic information from the image data source and downstream utilization element charateristic information from the downstream utilization element, said source characteristic information and downstream utilization element characteristic information defining at least one selected characteristic of said image data source and said downstream utilization element, and generating processing operation parameter information in response to the image processing selection information, the source characteristic information, the downstream utilization element characteristic information, and predetermined psychovisual attributes, said processing operation parameter information enabling an image data processing section to generate the psychovisually optimized processed image data; and

B. performing said at least one image processing operation in the image data processing section to generate a processed image from said psychovisually optimized processed image data in response to said processing operation parameter information.

16. The image processing method of claim 15 in which a series of said at least one image processing operations are performed in an order as selected by the image processing selection information.

17. The image processing method of claim 16 in which at least one of said series of said at least one image processing operation is performed using the processing operation parameter information.

18. The image processing system of claim 17 in which said at least one image processing operation is grouped into a predefined image processing operation sequence in accordance with said image processing selection information.

19. The image processing method of claim 18 in which the predefined image processing operation sequence is grouped into a sequence class, said predefined image processing operation sequence in each said sequence class defining one of a number of selected types of image processing operations, the predefined image processing operation sequence to be performed in each said sequence class in response to the image processing selection information.

20. The image processing method of claim 19 in which one said sequence class is a data decompression operation class including a plurality of said predefined image processing operation sequences each performing image data decompression in accordance with a selected decompression methodology, the image processing selection information including decompression methodology information to identify said predefined image processing operation sequence.

21. The image processing method of claim 19 in which one said sequence class is an image integrity operation class including a plurality of said predefined image processing operation sequences each performing said series of said at least one image processing operations in connection with spatial image information, the image processing selection information including image integrity selection information to identify said predefined image processing operation sequence related to image operations.

22. The image processing method of claim 19 in which one said sequence class is a special effects operation class including a plurality of said predefined image processing operation sequences each performing said series of said at least one image processing operations in relation to special effects, the image processing selection information including special effects selection information to identify said predefined image processing operation sequence related to special effects.

23. The image processing method of claim 19 in which one said sequence class is a data compression operation class including a plurality of said predefined image processing operation sequences each performing image data compression in accordance with a selected compression methodology, the image processing selection information including compression methodology information to identify said predefined image processing operation sequence.

24. The image processing method of claim 15 in which said step 15A includes:

A. generating trial parameter values;

B. generating a reference image quality value from both the source characteristic information defining image processing by the image data source, and the downstream utilization element characteristic information, in response to reference image information and the trial parameter values;

C. in a series of iterations, controlling both the trial parameter values and the reference image quality value in response to the processing operation parameter information to generate said trial parameter values for providing an optimal value of said reference image quality value.

25. The image processing method of claim 24 in which the generating step 24B comprises:

A. performing a processing operation corresponding to said at least one image processing operation in response to both the source characteristic information and the downstream utilization element characteristic information to generate a reference image profile value; and

B. generating the reference image quality value in response to the reference image profile value.

26. The image processing method of claim 25 in which both the source characteristic information and the downstream utilization element characteristic information include modulation transfer function information and noise power spectrum information, said method further comprising:

generating a modulation transfer function profile value from said modulation transfer function information in step 25A;

generating a noise power spectrum profile value from said noise power spectrum information in step 25A; and

generating said reference image quality value in step 25B in response to both said modulation transfer function profile value and the noise power spectrum profile value.

27. The image processing method of claim 26 wherein said step of generating said reference image quality value further comprises:

generating a sharpness quality value in response to both the modulation transfer function profile value and a spectral weighting factor; and

generating said reference image quality value in response to the sharpness quality value.

28. The image processing method of claim 26 wherein said step of generating said reference image quality value further comprises generating a graininess value in response to both the noise power spectrum profile value and a spectral weighting factor, and generating said reference image quality value from the graininess value.

29. The image processing method of claim 15 in which said image processing selection information is provided from memory or from an operator.

30. An image processing system for receiving input image data defining an input image from image source, processing the input image data in accordance with at least one image processing operation, and generating output image data corresponding to an output image in which predetermined psychovisual attributes have been automatically optimized for a select downstream application and the at least one image processing operation, said system comprising:

means for acquiring the input image data formatted as a plurality of input image data points corresponding, respectively, to pixels of the input image and objective metrics correlated with the predetermined psychovisual attributes of the input image;

a characteristic processing section for utilizing said objective metrics to evaluate an effect that the at least one image processing operation and the select downstream application will have on said objective metrics of the input image data, automatically selecting an optimal application of the at least one image processing operation, and generating processing parameter information for dictating a sequence and degree to which said optimal application of the at least one image processing operation is to be applied to the input image data; and

an image data processing section for receiving the input image data and said processing parameter information and in response thereto applying the at least one image processing operation to the input image data in accordance with said image processing parameter information to transform the input image data into the output image data for constructing the output image having optimal perceived quality.

31. An image process method for receiving input image data defining an input image from an image source, processing the input image data in accordance with at least one image processing operation, and generating output image data defining an output image in which predetermined psychovisual attributes have been automatically optimized for a select downstream application and the at least one image processing operation, said method comprising the steps of:

acquiring the image data formatted as a plurality of input image data points corresponding, respectively, to pixels of the input image and objective metrics con dated with the predetermined psychovisual attributes of the input image;

evaluating an effect that the at least one image processing operation and the select downstream application will have on said objective metrics of the input image data;

automatically selecting an optimal application of the at least one image processing operation;

generating processing parameter information for dictating a sequence and degree to which the at least one image processing operation is to be applied to the input image data; and

applying the at least one image processing operation to the input image data in accordance with said processing parameter information to transform the input image data into the output image data for constructing the out image having optimal/perceived image quality.