Image composition evaluating apparatus, information processing apparatus and methods thereof

• US 10,204,271 B2
• Filed: 07/31/2014
• Issued: 02/12/2019
• Est. Priority Date: 08/02/2013
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Lv, You, Jiang, Yong, Wu, Bo, Li, Xian
• Primary Examiner(s): Thomas, Mia M
• Application Number:
  US14/448,968
• Publication Number:
  US 20150036921A1
• Time in Patent Office: 1,657 Days
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

In the present invention, an attribution is extracted from each region obtained by segmentation of an image, relationships among the regions are described, and a composition of the image is evaluated based on the attributions and the relationships.

Claims

1. An apparatus for evaluating image composition comprising:

one or more memories that store a set of instructions; and

one or more processors that execute the set of instructions to;

segment an image into a plurality of regions;

extract at least one feature from each of the plurality of regions;

classify each of the plurality of regions into a preset class based on the extracted at least one feature and a trained model;

extract at least one attribution from each of the plurality of regions;

collect the at least one attribution for each of the plurality of regions;

merge isolated regions in the plurality of regions and update the attributions of the merged regions after collecting the at least one attribution for each of the plurality of regions;

describe relationships among the plurality of regions based on the extracted attributions; and

evaluate a composition of the image to determine whether at least one composition problem is included in the image, based on the extracted attributions, the described relationships and at least one preset criterion,

wherein when updating the attributions of the merged regions, if a plurality of regions merged into one region have more than one class, the class for the one merged region is the one that most of the plurality of regions have.

2. The apparatus according to claim 1, wherein an over-segmentation method is employed to segment the image into the plurality of regions.

3. The apparatus according to claim 2, wherein the over-segmentation method is a Felzenszwalb method or an SLIC method.

4. The apparatus according to claim 1, wherein a Recursive Neural Network method is employed to classify each of the plurality of regions into the preset class.

5. The apparatus according to claim 1, wherein the one or more processors execute the set of instructions to smooth borders between the regions after merging isolated regions and before updating the attributions of the regions.

6. The apparatus according to claim 1, wherein the one or more processors execute the set of instructions to:

calculate relative positions of the plurality of regions based on distances, neighbourhood and symmetries; and

calculate coordinate degrees among the plurality of regions based on relative area proportions, color matching and region complexity.

7. The apparatus according to claim 1, wherein the one or more processors execute the set of instructions to find locations and reasons for the at least one composition problem in the image with respect to the at least one preset criterion.

8. The apparatus according to claim 1, wherein the one or more processors execute the set of instructions to:

output and inform of the at least one composition problem after evaluating the composition of the image based on the extracted attributions, the described relationships and the at least one preset criterion.

9. The apparatus according to claim 1, wherein the image is a photographic image.

10. The apparatus according to claim 1, wherein the extracted attributions and the described relationships depend on the at least one preset criterion.

11. The apparatus according to claim 1, wherein the extracted attributions comprise at least one of class, position, size, color and texture.

12. The apparatus according to claim 1, wherein the described relationships comprise at least one of relative position, relative area proportion, color matching and texture matching.

13. The apparatus according to claim 1, wherein the at least one preset criterion comprises at least one of object on head, area size judgement, major object position judgement, kissing border, close to border, middle separation and color matching.

14. The apparatus according to claim 1, wherein the at least one composition problem includes at least one of an object on head, area size judgement, major object position judgement, kissing border, close to border, middle separation, and color matching.

15. A method for evaluating image composition comprising:

segmenting an image into a plurality of regions;

extracting at least one feature from each of the plurality of regions;

classifying each of the plurality of regions into a preset class based on the extracted at least one feature and a trained model;

extracting at least one attribution from each of the plurality of regions;

collecting the at least one attribution for each of the plurality of regions;

merging isolated regions in the plurality of regions;

updating the attributions of the merged regions after collecting the at least one attribution for each of the plurality of regions, wherein when updating the attributions of the merged regions, if a plurality of regions merged into one region have more than one class, the class for the one merged region is the one that most of the plurality of regions have;

describing relationships among the plurality of regions based on the extracted attributions; and

evaluating a composition of the image to determine whether at least one composition problem is included in the image, based on the extracted attributions, the described relationships and at least one preset criterion.

16. A non-transitory computer-readable medium storing instructions, which when executed by one or more processors, cause the one or more processors to perform a method comprising:

segmenting an image into a plurality of regions;

extracting at least one feature from each of the plurality of regions;

classifying each of the plurality of regions into a preset class based on the extracted at least one feature and a trained model;

extracting at least one attribution from each of the plurality of regions;

collecting the at least one attribution for each of the plurality of regions;

merging isolated regions in the plurality of regions;

updating the attributions of the merged regions after collecting the at least one attribution for each of the plurality of regions, wherein when updating the attributions of the merged regions, if a plurality of regions merged into one region have more than one class, the class for the one merged region is the one that most of the plurality of regions have;

describing relationships among the plurality of regions based on the extracted attributions; and

evaluating a composition of the image to determine whether at least one composition problem is included in the image, based on the extracted attributions, the described relationships and at least one preset criterion.