Discriminative embedding of local color names for object retrieval and classification

• US 9,600,738 B2
• Filed: 04/07/2015
• Issued: 03/21/2017
• Est. Priority Date: 04/07/2015
• Status: Expired due to Fees

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Murray, Naila, Perronnin, Florent C.
• Primary Examiner(s): Niu, Feng
• Application Number:
  US14/680,490
• Publication Number:
  US 20160300118A1
• Time in Patent Office: 714 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 10/7753 Incorporation of unlabelled data, e.g. multiple instance learning [MIL]

Abstract

A system and method enable similarity measures to be computed between pairs of images and between a color name and an image in a common feature space. Reference image representations are generated by embedding color name descriptors for each reference image in the common feature space. Color name representations for different color names are generated by embedding synthesized color name descriptors in the common feature space. For a query including a color name, a similarity is computed between its color name representation and one or more of the reference image representations. For a query which includes a query image, a similarity is computed between a representation of the query image and one or more of reference image representations. The method also enables combined queries which include both a query image and a color name to be performed. One or more retrieved reference images, or information based thereon, is then output.

Claims

1. A retrieval method comprising:

a) for each of at least one reference image, providing a reference image representation which has been generated by embedding color name descriptors for patches of the reference image into a feature space, each color name descriptor including an element for each of the color names in the set of color names;

b) for each of at least one color name in a set of color names, providing a color name representation which has been generated by embedding a synthesized color name descriptor in the same feature space as the reference images, each synthesized color name descriptor including an element for each of the color names in the set of color names, each embedded synthesized color name descriptor having a same number of dimensions as each of the reference image representations;

c) receiving a query comprising at least one of;

a color name selected from the set of color names, and

a query image;

d) for a query which includes a color name, retrieving the respective color name representation;

e) for a query which includes a query image, providing a query image representation which has been generated by embedding color name descriptors for the query image in the multidimensional feature space;

f) computing a similarity between;

at least one of the at least one reference image representations, and

at least one of the color name representation and the query image representation; and

g) outputting information based on the computed similarity,

wherein at least one of a), b), c), d), e), f), and g) is performed with a processor.

2. The method of claim 1, wherein the provided color name representation has been generated by:

synthesizing a local color name descriptor;

generating a color name representation comprising embedding the synthesized local color name descriptor with a first embedding function.

3. The method of claim 2, wherein the synthesized local color name descriptor includes a non-zero value for only one of a plurality of elements, each element corresponding to a color name in the set of color names.

4. The method of claim 2, wherein the first embedding function is a Fisher Vector embedding function.

5. The method of claim 1, wherein the provided reference image representation has been generated by:

assigning to each of a set of pixels of the reference image, a probability distribution over a predefined set of color names;

for each of a plurality of patches of the reference image;

generating a local color name descriptor, comprising aggregating the pixel probability distributions, and

generating an embedded patch descriptor comprising embedding the local color name descriptor with a second embedding function; and

generating the reference image representation, comprising aggregating the embedded patch descriptors.

6. The method of claim 5, wherein the second embedding function is a Fisher Vector embedding function.

7. The method of claim 5, wherein the reference image representation has been generated by:

defining a plurality of patches of the reference image, each patch including a plurality of cells;

for pixels of each cell aggregating the pixel probability distributions; and

for each of the plurality of patches of the reference image, generating the local color name descriptor, comprising concatenating the aggregated pixel probability distributions of the cells in the patch.

8. The method of claim 1, wherein the query includes a query image and the generating of the query image representation comprises:

assigning to each of a set of pixels of the query image, a probability distribution over the set of color names;

for each of a plurality of patches of the query image;

generating a local color name descriptor, comprising aggregating the pixel probability distributions, and

generating an embedded patch descriptor comprising embedding the local color name descriptor with a second embedding function; and

generating a query image representation, comprising aggregating the embedded patch descriptors.

9. The method of claim 8, wherein the query image representation is generated by:

defining a plurality of patches of the query image, each patch including a plurality of cells;

for each cell, aggregating the pixel probability distributions of the pixels of the cell; and

for each of the plurality of patches of the query image, generating the local color name descriptor, comprising concatenating the aggregated pixel probability distributions of the cells in the respective patch.

10. The method of claim 9, wherein the aggregating of the pixel probability distributions is performed by max pooling aggregation.

11. The method of claim 1, wherein the reference image comprises a plurality of reference images.

12. The method of claim 1, wherein the at least one color name includes at least four color names.

13. The method of claim 1, wherein the at least one color name includes no more than twenty color names.

14. The method of claim 1, wherein the received query includes a color name and the similarity is computed based on the color name representation.

15. The method of claim 14, wherein the received query further includes a query image, and the similarity is computed based on the query image representation and the color name representation.

16. The method of claim 1, wherein the computing of the similarity comprises computing a cosine similarity between the at least one of the at least one reference image representation, and the at least one of the color name representation and the query image representation.

17. A computer program product comprising non-transitory memory storing instructions which when executed by a computer, perform the method of claim 1.

18. A system comprising memory which stores instructions for performing the method of claim 1 and a processor in communication with the memory for executing the instructions.

19. A retrieval method comprising:

a) for each of at least one reference image, providing a reference image representation which has been generated by embedding color name descriptors for the reference image in a feature space;

b) for each of at least one color name in a set of color names, providing a color name representation which has been generated by embedding a synthesized color name descriptor in the same feature space as the embedded color name descriptors of the reference image;

c) providing for receiving a query comprising at least one of;

a color name selected from the set of color names, and

a query image;

d) for a query which includes a color name, retrieving the respective color name representation;

e) for a query which includes a query image, providing a query image representation which has been generated by embedding color name descriptors for the query image in the feature space;

f) computing a similarity between;

at least one of the at least one reference image representations, and

at least one of the color name representation and the query image representation,

wherein when the received query includes both a color name selected from the set of color names and a query image, the similarity is computed based on the query image representation and the color name representation; and

g) outputting information based on the computed similarity,

wherein at least one of a), b), c), d), e), f), and g) is performed with a processor.

20. A system comprising:

an image representation generation component which generates representations of images by embedding color name descriptors for the image in a feature space;

memory which stores a color name representation for each of a set of color names, generated by embedding a synthesized color name descriptor in the feature space into which the images are embedded, the embedded synthesized color name descriptor having a same number of dimensions as each of the image representations;

a comparison component which receives a query comprising at least one of a query image and one of the set of color names and computes a similarity between the query and at least one reference image, wherein in computing the similarity, the at least one reference image is embedded in the feature space and the at least one of the query image and one of the set of color names is embedded in the feature space, wherein when the query includes both a color name selected from the set of color names and a query image, the similarity is computed based on the query image representation and the color name representation;

an output component which outputs information based on the computed similarity; and

a processor which implements the image representation generation component, the comparison component, and the output component.

21. A method for generating a system for comparing images and color names in a common multidimensional feature space comprising:

for each of a set of color names, generating a color name representation in a multidimensional feature space, comprising;

synthesizing a local color name descriptor; and

generating a color name representation comprising embedding the synthesized local color name descriptor in a multidimensional feature space with a first embedding function;

for each of a set of reference images, generating a reference image representation in the same multidimensional feature space as the color name representations, comprising;

assigning to each of a set of pixels of the reference image, a probability distribution over a predefined set of color names;

for each of a plurality of patches of the reference image;

generating a local color name descriptor, comprising aggregating the pixel probability distributions, and

generating an embedded patch descriptor comprising embedding the local color name descriptor with a second embedding function; and

aggregating the embedded patch descriptors to generate the reference image representation;

storing the color name representations and reference image representations;

storing instructions for;

receiving a query comprising at least one of;

a color name selected from the set of color names, and

a query image, and

computing a similarity between the query and at least one of the stored reference image representations, wherein in computing the similarity, the at least one of the query image and one of the set of color names is embedded in the multidimensional feature space; and

wherein at least one of the generating of the color name representations and the generating of the reference image representations is performed with a processor.