LABEL-EMBEDDING VIEW OF ATTRIBUTE-BASED RECOGNITION

• US 20140376804A1
• Filed: 06/21/2013
• Published: 12/25/2014
• Est. Priority Date: 06/21/2013
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Akata, Zeynep, Perronnin, Florent C., Harchaoui, Zaid, Schmid, Cordelia L.
• Granted Patent: US 10,331,976 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06F 18/2132 based on discrimination criteria, e.g. discriminant analysis
• CPC Class Codes: G06F 18/21345 enforcing sparsity or involving a domain transformation
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06F 2218/00 Aspects of pattern recognition specially adapted for signal processing
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods

Abstract

In image classification, each class of a set of classes is embedded in an attribute space where each dimension of the attribute space corresponds to a class attribute. The embedding generates a class attribute vector for each class of the set of classes. A set of parameters of a prediction function operating in the attribute space respective to a set of training images annotated with classes of the set of classes is optimized such that the prediction function with the optimized set of parameters optimally predicts the annotated classes for the set of training images. The prediction function with the optimized set of parameters is applied to an input image to generate at least one class label for the input image. The image classification does not include applying a class attribute classifier to the input image.

Claims

1. A non-transitory storage medium storing instructions readable and executable by an electronic data processing device to perform a method including the operations of:

representing classes of a set of classes Y={y_j, j=1, . . . , C} by class attribute vectors φ

(y_j) where φ

is an embedding function that embeds a class in an attribute space of dimensionality E where each dimension a_i,i=1, . . . , E of the attribute space corresponds to a class attribute;

representing training images x_n of a set of training images S labeled by respective training image class labels y_n as θ

(x_n) where θ

is an embedding function that embeds an image in an image feature space of dimensionality D;

optimizing respective to the set of training images S a set of parameters w of a prediction function

2. The non-transitory storage medium as set forth in claim 1 wherein the applying does not include learning image attribute classifiers.

3. The non-transitory storage medium as set forth in claim 1 wherein the class attributes of the attribute space include image attributes.

4. The non-transitory storage medium as set forth in claim 3 wherein the set of classes Y comprises a hierarchy of classes and the class attributes of the attribute space further include class hierarchy attributes derived from the hierarchy of classes.

5. The non-transitory storage medium as set forth in claim 1 wherein the set of classes Y comprises a hierarchy of classes and the class attributes of the attribute space include class hierarchy attributes derived from the hierarchy of classes.

6. The non-transitory storage medium as set forth in claim 1 wherein the optimizing and applying operations include:

computing F(x,y;

w) as F(x,y;

W)=θ

(x)′

Wφ

(y) where W is a D×

E matrix representing the set of parameters w of the prediction function ƒ

(x;

w).

7. The non-transitory storage medium as set forth in claim 6 wherein the applying further includes:

generating class attribute scores for the input image for class attributes of the attribute space based on θ

(x_in)′

W where x_in denotes the input image.

8. The non-transitory storage medium as set forth in claim 1 wherein the optimizing includes optimizing at least one of a ranking objective, a multiclass objective, and a regression objective.

9. The non-transitory storage medium as set forth in claim 1 wherein the stored instructions are further readable and executable by the electronic data processing device to perform at least one operation of a group consisting of:

cataloging the input image using the at least one class label for the input image;

retrieving images from an images database having the same or similar labeling to the at least one class label for the input image; and

annotating the input image using the at least one class label for the input image wherein the input image is embedded in a multi-modality document that includes text and images including the input image.

10. A method comprising:

embedding each class of a set of classes in an attribute space where each dimension of the attribute space corresponds to a class attribute, the embedding generating a class attribute vector for each class of the set of classes;

optimizing a set of parameters of a prediction function operating in the attribute space respective to a set of training images annotated with classes of the set of classes such that the prediction function with the optimized set of parameters optimally predicts the annotated classes for the set of training images; and

applying the prediction function with the optimized set of parameters to an input image to generate at least one class label for the input image;

wherein the method is performed by an electronic data processing device.

11. The method of claim 10 wherein the class attributes of the attribute space include image attributes that are semantically meaningful.

12. The method of claim 10 wherein the class attributes of the attribute space belong to the group consisting of image attributes that are semantically meaningful and class hierarchy attributes derived from a semantic hierarchy of the set of classes.

13. The method of claim 10 wherein the method is performed by the electronic data processing device further comprises:

embedding images of the set of training images in an image features vector space where each dimension of the image features vector space corresponds to an image feature;

the prediction function operating in the image feature vector space and in the attribute space.

14. The method of claim 13 wherein the set of parameters comprises a parameters matrix whose matrix elements relate image features of the image features vector space with class attributes of the attribute space.

15. The method of claim 14 wherein the method is performed by the electronic data processing device further comprises:

embedding the input image in the image features vector space to generate an input image features vector representing the input image;

wherein the applying comprises multiplying the input image features vector and the parameters matrix, said multiplying also yielding class attribute scores for class attributes of the attribute space.

16. The method of claim 10 wherein the method is performed by the electronic data processing device does not include applying a class attribute classifier to the input image.

17. An apparatus comprising:

an electronic data processing device programmed to perform a method including the operations of;

embedding classes of a set of classes Y={y_j, j=1, . . . , C} in an attribute space of dimensionality E where each dimension a_i, i=1, . . . , E of the attribute space corresponds to a class attribute;

embedding training images x_n of a set of training images S labeled by respective training image class labels y_n in an image feature space of dimensionality D;

optimizing a set of parameters w of a prediction function y=ƒ

(x;

w) operating in the image feature space and in the attribute space respective to the set of training images S wherein x denotes an image and y denotes the predicted class label for the image x; and

applying the prediction function y=ƒ

(x;

w) with the optimized set of parameters w to an input image to generate at least one class label for the input image.

18. The apparatus of claim 17 wherein the prediction function is y=ƒ

(x;

w)=argmax<sub>yε

Y</sub> {θ

(x)′

Wφ

(y)} where θ

is an embedding function used in embedding the training images, φ

is an embedding function used in embedding the classes, and W is a parameters matrix representing the set of parameters w.

19. The apparatus of claim 17 wherein each dimension a_i, i=1, . . . , E of the attribute space corresponds to a class attribute comprising an image attribute.

20. The apparatus of claim 17 wherein each dimension a_i, i=1, . . . , E of the attribute space corresponds to a class attribute comprising one of an image attribute and a class hierarchy attribute.