Image processing apparatus, method thereof, program, and image capturing apparatus

• US 8,861,881 B2
• Filed: 08/11/2011
• Issued: 10/14/2014
• Est. Priority Date: 08/18/2010
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Tate, Shunta, Takimoto, Masafumi, Mitarai, Yusuke, Mori, Katsuhiko, Matsugu, Masakazu
• Primary Examiner(s): TSAI, TSUNG YIN
• Application Number:
  US13/207,544
• Publication Number:
  US 20120045095A1
• Time in Patent Office: 1,160 Days
• Field of Search: None
• US Class Current: 382/254
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/75 involving models

Abstract

An image processing apparatus stores model information representing a subject model belonging to a specific category, detects the subject from an input image by referring to the model information, determines a region for which an image correction is to be performed within a region occupied by the detected subject in the input image, stores, for a local region of the image, a plurality of correction data sets representing correspondence between a feature vector representing a feature before correction and a feature vector representing a feature after correction, selects at least one of the correction data sets to be used to correct a local region included in the region determined to undergo the image correction, and corrects the region determined to undergo the image correction using the selected correction data sets.

Claims

1. An image processing apparatus comprising:

a model storage unit configured to store model information representing a subject model belonging to a specific category, the model information including location information of a part of the subject model;

a detection unit configured to detect a first region of the subject from an input image by referring to the model information and to detect a second region of the part of the subject, corresponding to the part of the subject model, in the first region by referring to the model information;

a determination unit configured to determine a partial image for which an image correction is to be performed, the partial image including at least the detected first region of the subject in the input image;

a correction data set storage unit configured to store a plurality of correction data sets, each of the plurality of correction data sets including a plurality of first partial images being based on an image including a subject and feature vectors of second partial images being generated by processing the first partial images;

a selection unit configured to select, from said correction data set storage unit, at least one of the correction data sets on the basis of relative position between the partial image determined by said determination unit and the second region;

an extraction unit configured to extract a feature vector from the partial image determined by said determination unit;

a calculation unit configured to calculate degrees of similarities between the feature vector extracted by said extraction unit and feature vectors of the second partial images in the selected correction data set; and

a correction unit configured to select at least one feature vector of the second partial image on the basis of the calculated degrees of similarities and correct the partial image determined by said determination unit using the first partial image corresponding to the selected feature vector of the second partial image.

2. The apparatus according to claim 1, wherein said correction unit performs correction of a background region, a region of the part of the subject, and a region of a subject other than the part using different correction data sets.

3. The apparatus according to claim 1, wherein

said model storage unit stores the model information for each subcategory representing a more detailed classification of the category,

the apparatus further comprises a discrimination unit configured to discriminate a subcategory of the subject detected by said detection unit, and

said selection unit selects the correction data sets based on a discrimination result of the subcategory.

4. The apparatus according to claim 3, wherein the correction data sets stored in said correction data set storage unit include correction data sets common to all subjects belonging to a single category and correction data sets that are different for each subcategory.

5. The apparatus according to claim 1, wherein the first partial image is a high-resolution image and the second partial image is a low-resolution image.

6. The apparatus according to claim 1, wherein the first partial image is a high-dynamic range image and the second partial image is a low-dynamic range image.

7. The apparatus according to claim 1, wherein the first partial image is a blurred image and the second partial image is a sharp image.

8. The apparatus according to claim 1, wherein the first partial image is a monochrome image and the second partial image is a colored image.

9. The apparatus according to claim 1, wherein the first partial image is a toneless image and the second partial image is a bright colored image.

10. The apparatus according to claim 1, wherein the first partial image is an image captured with ambient light and the second partial image is an image captured with using flush light.

12. A non-transitory computer-readable storage medium storing a program that causes a computer to function as each unit of an image processing apparatus of claim 1.

13. An image capturing apparatus comprising:

an image capturing unit adapted to capture an image; and

an image processing apparatus of claim 1 configured to process an input image that is the image captured by said image capturing unit.

11. An image processing method of an image processing apparatus including a model storage unit configured to store model information representing a subject model belonging to a specific category, the model information including location information of a part of the subject model, and a correction data set storage unit configured to store a plurality of correction data sets, each of the plurality of correction data sets including a plurality of first partial images being based on an image including a subject and feature vectors of second partial images being generated by processing the first partial images, the method comprising:

detecting a first region of the subject from an input image by referring to the model information;

detecting a second region of the part of the subject, corresponding to the part of the subject model, in the first region by referring to the model information;

determining a partial image for which an image correction is to be performed, the partial image including at least the detected first region of the subject in the input image;

selecting, from the correction data set storage unit, at least one of the correction data sets on the basis of relative position between the partial image determined by the determining step and the second region;

extracting a feature vector from the partial image determined in said determining step;

calculating degrees of similarities between the feature vector extracted in the extracting step and feature vectors of the second partial images in the selected correction data set;

selecting at least one feature vector of the second partial image on the basis of the calculated degrees of similarities; and

correcting the partial image determined in said determining step using the first partial image corresponding to the selected feature vector of the second partial image.