IMAGE RANKING BASED ON ABSTRACT CONCEPTS

• US 20120163710A1
• Filed: 12/22/2010
• Published: 06/28/2012
• Est. Priority Date: 12/22/2010
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Marchesotti, Luca, Csurka, Gabriela, SKAFF, SANDRA
• Granted Patent: US 8,532,377 B2
• Time in Patent Office: Days
• US Class Current: 382/165
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene

Abstract

A system and method for ranking images are provided. The method includes receiving a query comprising a semantic part and an abstract part, retrieving a set of images responsive to the semantic part of the query, and computing first scores for the retrieved images in the set of retrieved images. The first score of an image can be based on a relevance of that image to the semantic part of the query (and not to the abstract part of the query). The method further includes identifying a chromatic concept model from a set of chromatic concept models. This identification can be based on the abstract part of the query (and not on the semantic part of the query). The chromatic concept model includes an optionally-weighted set of colors expressed in a perceptually uniform color space. For retrieved images in the set of retrieved images, the method includes computing a chromatic image model based on colors of the image, the chromatic image model comprising a weighted set of colors expressed in the perceptually uniform color space and computing a comparison measure between the chromatic image model and the chromatic concept model. The retrieved images are scored with respective second scores that are based on the computed comparison measures. The retrieved images are ranked based on a combined score for a respective retrieved image which is a function of the first and second scores.

Claims

1. A method for ranking images comprising:

receiving a query comprising a semantic part and an abstract part;

retrieving a set of images responsive to the semantic part of the query;

computing first scores for the retrieved images in the set of retrieved images, each first score being based on a relevance of a retrieved image to the semantic part of the query;

identifying one of a set of chromatic concept models, based on the abstract part of the query, the identified chromatic concept model comprising an optionally-weighted set of colors expressed in a perceptually uniform color space;

for retrieved images in the set of retrieved images;

computing a chromatic image model based on colors of the image, the chromatic image model comprising a weighted set of colors expressed in the perceptually uniform color space, and

computing a comparison measure between the chromatic image model and the identified chromatic concept model;

computing second scores for the retrieved images that are based on the computed comparison measures; and

ranking the retrieved images based on a combined score for a respective retrieved image which is a function of the first and second scores.

2. The method of claim 1, wherein at least one of the computing of the first scores, computing of the second scores, and ranking of the retrieved images is performed with a computer processor.

3. The method of claim 1, further comprising, providing for a user to review at least some of the retrieved images.

4. The method of claim 3, wherein the method includes displaying at least some of the retrieved images in an arrangement which reflects the combined scores.

5. The method of claim 3, further comprising visualizing at least one of:

the chromatic concept model; and

at least one of the chromatic image models.

6. The method of claim 1, wherein the perceptually uniform color space is one which has a lightness component and two chromatic components.

7. The method of claim 6, wherein the weighted set of colors of the chromatic image model are expressed only in the chromatic components of the perceptually uniform color space.

8. The method of claim 1, further comprising computing a chromatic concept model for each of a set of abstract concepts.

9. The method of claim 8, wherein the computing of the chromatic concept model for each of the set of abstract concepts comprises computing a Gaussian Mixture Model wherein observations for the Gaussian mixture model are colors of at least one color palette which is associated with the abstract concept.

10. The method of claim 1, wherein the computing of the chromatic image model comprises computing a Gaussian Mixture Model wherein observations for the Gaussian mixture model are colors of pixels of the image.

11. The method of claim 1, wherein the computing of the comparison measure between the chromatic image model and the chromatic concept model comprises computing a distance measure which is based on Euclidian distances in the perceptually uniform color space between the colors of the chromatic image model and colors of the chromatic concept model and the associated weights.

12. The method of claim 11, wherein the computing of the comparison measure comprises computing a Color-based Earth Mover'"'"'s Distance (CEMD) according to the expression:

13. The method of claim 12, wherein the computing of the comparison measure comprises minimizing:

C=Σ

_i=1^m Σ

_j=1ⁿ f_ij d_ij  

 

(3)

subject to at least one of;

f_ij≧

0 1≦

i≦

m, 1≦

j≦

n  

 

(3A)

Σ

_j=1ⁿ f_ij≦

w_S1i 1≦

i≦

m  

 

(3B)

Σ

_i=1^m f_ij≦

w_S2j 1≦

j≦

n  

 

(3C)

Σ

_i=1^m Σ

_j=1ⁿ f_ij=min(Σ

_i=1^m w_S1i, Σ

_j=1ⁿ w_S2j)  

 

(3D)

where w_S1i is the weight assigned to color S_1i and w_S2j is the weight assigned to color s_2j of S₂.

14. The method of claim 1, wherein the computing a combined score for retrieved images in the set as a function of the first and second scores comprises computing a weighed combination of the first and second scores.

15. The method of claim 1, further comprising displaying the images in a first arrangement based on the first scores and displaying the images in a second arrangement based on the combined scores.

16. The method of claim 1, wherein the method further comprises converting colors of at least one of the retrieved images from an original color space to the perceptually uniform color space prior to computing the image'"'"'s chromatic image model.

17. A computer program product comprising a non-transitory recording medium encoding instructions which, when executed by a computer, perform the method of claim 1.

18. A system for performing the method of claim 1, comprising memory which stores instructions for performing the method and a processor in communication with the memory for executing the instructions.

19. A system for ranking images comprising:

a set of chromatic concept models stored in memory, each chromatic concept model comprising an optionally-weighted set of colors expressed in a perceptually uniform color space which represents an abstract concept;

a chromatic image component which computes a chromatic image model for each of a set of images retrieved in response to a semantic part of an input query, the chromatic image model being based on colors of the image, the chromatic image model comprising a weighted set of colors expressed in the perceptually uniform color space;

a scoring component which scores the retrieved images in the set of retrieved images, the score for each image being based on a relevance to the semantic part of an input query and a comparison measure between its computed chromatic image model and a selected one of the chromatic concept models; and

a visualization component which generates a visualization of the retrieved images based on their respective scores.

20. The system of claim 19, further comprising a graphical user interface configured for displaying at least some of the ranked images.

21. The system of claim 19, further comprising a chromatic concept component which models each of the chromatic concepts based on a respective set of color palettes.

22. A method for ranking images comprising:

providing a set of chromatic concept models in non-transitory memory, each of the chromatic concept models comprising an optionally-weighted set of colors expressed in a perceptually uniform color space which is representative of an abstract concept;

retrieving a set of images from a database, the retrieved images being responsive to a textual query;

for retrieved images in the set, computing a respective chromatic image model based on colors of the image, the chromatic image model comprising a weighted set of colors expressed in a perceptually uniform color space, and

scoring the retrieved images in the set of retrieved images with a combined score which is based on the relevance of the images to the textual query and a distance of the respective chromatic image model to a selected one of the chromatic concept models.

23. A method for generating a system for ranking images comprising:

for each of a set of abstract concepts, with an algorithm, generating a set of chromatic concept models, each of the chromatic concept models comprising an optionally-weighted set of colors expressed in a perceptually uniform color space which is representative of an abstract concept;

providing a component which generates a respective chromatic image model based on colors of an image, the chromatic image model comprising a weighted set of colors expressed in a perceptually uniform color space; and

providing a scoring component for scoring images retrieved in response to a query with a combined score which is based on the relevance of the images to a semantic part of the query and a distance of a respective chromatic image model to a selected one of the chromatic concept models.