CLASS-BASED IMAGE ENHANCEMENT SYSTEM

• US 20080317358A1
• Filed: 06/25/2007
• Published: 12/25/2008
• Est. Priority Date: 06/25/2007
• Status: Active Grant

• Current Assignee: ASML Netherlands BV (ASML Holding NV), Carl Zeiss AG (Carl-Zeiss-Stiftung), Tarsium B.V.
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Bressan, Marco, Csurka, Gabriela, Dance, Christopher R.
• Granted Patent: US 7,933,454 B2
• Time in Patent Office: Days
• US Class Current: 382/224
• CPC Class Codes: G06T 2207/20008 Globally adaptive
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables

Abstract

A method for image enhancement includes providing for a semantic class to be assigned to a digital image based on image content, the assigned semantic class being selected from a plurality of semantic classes. The method further includes providing for an aesthetic enhancement to be applied to the image based on image quality of the image and the assigned semantic class, the enhancement including at least one enhancement dimension selected from a plurality of enhancement dimensions.

Claims

1. A method for image enhancement, comprising:

providing for a semantic class to be assigned to a digital image based on image content, the assigned semantic class being selected from a plurality of semantic classes; and

providing for an aesthetic enhancement to be applied to the image based on an image quality of the image and the assigned semantic class, the enhancement including at least one enhancement dimension selected from a plurality of enhancement dimensions.

2. The method of claim 1, wherein at least one of the enhancement dimensions is associated with a plurality of enhancement modes whereby an enhancement is applied in a selected one of the modes depending on the assigned semantic class.

3. The method of claim 1, wherein the aesthetic enhancement is based on a confidence value representing a confidence with which the semantic class is assigned.

4. The method of claim 1, further comprising computing at least one image quality related parameter for an input digital image and determining the image quality therefrom.

5. The method of claim 1, wherein the semantic class is assigned independently of the image quality.

6. The method of claim 1, wherein the providing for the aesthetic enhancement to be applied comprises:

determining whether the image could be aesthetically enhanced based on the image quality and, where the determination indicates that the image could be aesthetically enhanced, determining a candidate enhancement including determining at least one enhancement dimension; and

where the enhancement dimension is applicable in a plurality of enhancement modes, selecting an enhancement mode from the plurality of enhancement modes based on the assigned semantic class and optionally on an assignment confidence value.

7. The method of claim 1, further comprising:

applying the candidate aesthetic enhancement in the selected enhancement mode to the image to generate a modified image.

8. The method of claim 1, further comprising:

rendering the modified image on an output device.

9. The method of claim 1, wherein the image quality includes at least one dimension selected from the group consisting of luminance, chrominance, noise, edge levels, and blocking artifacts.

10. The method of claim 1 wherein the plurality of semantic classes includes at least three semantic classes.

11. The method of claim 1, wherein the plurality of enhancement dimensions comprises at least three enhancement dimensions.

12. The method of claim 2, wherein at least one of the following applies:

a plurality of the enhancement dimensions is each associated with at least two enhancement modes; and

at least one of the enhancement dimensions is associated with at least three enhancement modes.

13. The method of claim 1, wherein the assigning of the semantic class to the image includes categorizing the image with at least one classifier which has been trained on a set of training images, each of the training images having been manually assigned to a class selected from a set of classes.

14. The method of claim 1, wherein the assigning the image to a semantic class includes:

identifying a plurality of patches in the input image;

extracting features from the patches;

assigning at least one visual word to the image based on the extracted features, each visual word being selected from a predefined vocabulary of visual words; and

assigning at least one of the semantic classes to the image based on the assigned at least one visual word; and

optionally, assigning a confidence value to each of the at least one semantic classes based on the assigned visual words.

15. The method of claim 14, wherein the extracting of features from the patches includes computing a features vector for each of the identified patches and the assigning of at least one visual word includes matching the features vector to a visual word in the vocabulary.

16. The method of claim 1, wherein at least one of the enhancement dimensions is selected from the group consisting of sharpening, exposure correction, color balance and saturation adjustment, contrast enhancement, edge enhancement, blocking artifact reduction, noise reduction, and combinations thereof.

17. The method of claim 1, wherein the providing for an aesthetic enhancement to be applied to the image includes inputting the image quality and assigned semantic class into a model which maps image quality dimensions and semantic classes to enhancements.

18. The method of claim 17, wherein in the model, for a first of the enhancement dimensions, the enhancement mode is a default mode for a first of the semantic classes and is a non-default mode for a second of the semantic classes

19. The method of claim 1, wherein at least one of the plurality of semantic classes is selected from the group consisting of a flower class, an urban class, and a special effects class and, optionally, when one of the classes is a special effects class, no enhancement is made.

20. The method of claim 2, wherein the enhancement modes are selected from a set of discrete enhancement modes in which some of the enhancement modes are default modes and others of the enhancement modes provide for an increased enhancement as compared with the default mode for that enhancement.

21. A computer program product which encodes instructions which, when executed on a computer, performs the method of claim 1.

22. An image processing system which executes instructions for performing the method of claim 1.

23. An image enhancement system comprising:

an image quality component for assigning an image quality to the images, based on image quality parameters of input digital images;

an image categorizing component for assigning a semantic class to an input image based on image content; and

a model which maps assigned image quality and assigned semantic class to candidate aesthetic enhancements, whereby for at least some of the input images, the enhancement is applied in a mode which is dependent on the assigned semantic class.

24. The image enhancement system of claim 23, further comprising:

an image enhancement component which accesses the data structure to determine a candidate enhancement and enhancement mode based on the class and image quality assigned to the image.

25. The image enhancement system of claim 23, wherein the image categorizing component includes at least one classifier which classifies an image based on automatically identified features for each of a plurality of patches of the image.

26. The image enhancement system of claim 23, wherein the image quality component analyses the image quality for assigning the image quality related parameters to the images.

27. A method for forming an image enhancement system, comprising:

identifying a set of candidate aesthetic enhancement dimensions for applying to digital images based on a measure of image quality;

for at least some of the candidate enhancement dimensions, establishing a plurality of discrete modes in which the enhancement dimensions are to be applied;

identifying a set of semantic classes to which digital images are to be assigned by an automated categorizer, based on their image content; and

mapping the semantic classes and image quality measures to discrete modes of the candidate aesthetic enhancement dimensions.

28. The method of claim 27, further comprising:

assigning one of the classes to each of the images in a training set based on image content;

selecting at least one candidate enhancement dimension for each image in the training set;

rendering each of the images in a plurality of enhancement modes for each of the selected candidate enhancement dimensions;

and wherein the mapping is based on user preferences among the rendered images.

29. An image processing system which takes as input a digital image and determines a candidate enhancement for the image based on image quality for the image, the image processing system determining whether the image should be assigned to a special effects class and, where the image is assigned to the special effects class, determines that the candidate enhancement is not to be applied.