Ranking approach to train deep neural nets for multilabel image annotation

• US 9,552,549 B1
• Filed: 07/28/2014
• Issued: 01/24/2017
• Est. Priority Date: 07/28/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Jia, Yangqing, Gong, Yunchao, Leung, King Hong Thomas, Toshev, Alexander Toshkov, Ioffe, Sergey
• Primary Examiner(s): Hill, Stanley K
• Assistant Examiner(s): Misir, Dave
• Application Number:
  US14/444,272
• Time in Patent Office: 911 Days
• Field of Search: 706/25
• US Class Current: 1/1
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

Systems and techniques are provided for a ranking approach to train deep neural nets for multilabel image annotation. Label scores may be received for labels determined by a neural network for training examples. Each label may be a positive label or a negative label for the training example. An error of the neural network may be determined based on a comparison, for each of the training examples, of the label scores for positive labels and negative labels for the training example and a semantic distance between each positive label and each negative label for the training example. Updated weights may be determined for the neural network based on a gradient of the determined error of the neural network. The updated weights may be applied to the neural network to train the neural network.

Claims

1. A method performed by one or more computers, the method comprising:

receiving respective label scores determined by a neural network for each of at least two labels for at least one training example, wherein at least one of the at least two labels for each training example is a positive label for the training example and at least one other of the at least two labels for each training example is a negative label for the training example;

determining an error of the neural network based on a semantic ranking loss of the label scores, wherein the semantic ranking loss is determined according to;

2. The method of claim 1, wherein D(y_c+j, y_c−

_k) is determined based on number of nodes traversed to travel between a leaf for the positive label y_c+j and a leaf for the negative label y<sub>c−

k</sub> in a semantic tree.

3. The method of claim 1, wherein determining the updated weights of the neural network comprises training the neural network with gradient descent backpropagation.

4. The method of claim 1, wherein the at least one training example is an image, and wherein the at least two labels are each respective words or phrases.

5. The method of claim 1, wherein a label is a positive label for one of the at least one training example when the label has been predetermined to describe the content of the training example, and wherein a label is a negative label for the one of the at least one training example when the label has been predetermined to not describe the content of the training example.

6. The method of claim 1, wherein determining the updated weights for the neural network comprises attempting to minimize J.

7. The method of claim 1, wherein there are two or more positive labels for each of the at least one training example.

8. The method of claim 1, further comprising:

receiving an image;

receiving a label corpus comprising labels;

scoring, by the neural network, each label from the label corpus with a score representing how well the label describes the image; and

annotating the image with a predetermined number of the labels with highest scores from the label corpus.

9. A system for a ranking approach to training deep neural networks for multilabel image annotation, comprising:

one or more computers;

storage coupled to the one or more computers on which is stored a training data set including training examples, a label corpus, and a semantic structure; and

a machine learning system deployed on the one or more computers, the machine learning system comprising a neural network and a neural network trainer,

the neural network adapted to receive the label corpus and training examples from the training data set, generate respective label scores for each of at least two labels in the label corpus for at least one training example from the training data set, and receive updated weights, and

the neural network trainer adapted to determine an error of the neural network based on a semantic ranking loss of the label scores and to determine the semantic ranking loss according to;

J=Σ

_i=1ⁿΣ

_j=1^c+Σ

_k=1^c−

D(y_c+j,y<sub>c−

k</sub>)max(0,ρ

−

x_iW_c+j+x_iW<sub>c−

k</sub>)

where W is a ranking function of the neural network, n is the number of training examples, x_i is an ith training example, c+ is the number of positive labels for the training example x_i, c−

is the number of negative labels for the training example x_i, ρ

is a margin for hinge loss, y_c+j is the jth positive label, y<sub>c−

k</sub> is kth negative label, D(y_c+j, y<sub>c−

k</sub>) is a function that evaluates the semantic distance between two labels, y_c+j and y<sub>c−

k</sub>, x_iW_c+j is the label score given to the jth positive label when the ranking function W is used to evaluate the training example x_i, and x_iW<sub>c−

k</sub> is the label score given to the kth negative label when the ranking function W is used to evaluate the training example x_i.

10. The system of claim 9, wherein the neural network trainer is further adapted to determine D(y_c+j, y_c−

_k) based on number of nodes traversed to travel between a leaf for the positive label y_c+j and a leaf for the negative label y<sub>c−

k</sub> in a semantic tree.

11. The system of claim 9, wherein the neural network trainer is further adapted to determine the updated weights of the neural network by training the neural network with gradient descent backpropagation.

12. The system of claim 9, wherein the training data set comprises training example images.

13. The system of claim 9, wherein a label is a positive label for one of the at least one training example when the label has been predetermined to describe the content of the training example, and wherein a label is a negative label for the one of the at least one training example when the label has been predetermined to not describe the content of the training example.

14. The system of claim 9, wherein the neural network trainer is further adapted to determine the updated weights for the neural network by attempting to minimize J.

15. The system of claim 9, wherein there are two or more positive labels for each of the at least one training example from the training data set.

16. The system of claim 9, wherein the neural network is further adapted to receive an image, receive the label corpus comprising labels, score, by the neural network, each label from the label corpus as to how well the label describes the image;

and annotate the image with a predetermined number of the labels with highest scores from the label corpus.

17. A system comprising:

one or more computers and one or more storage devices storing instructions which are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving respective label scores determined by a neural network for at least two labels for at least one training example wherein at least one of the at least two labels for each training example is a positive label for the training example and at least one other of the at least two labels is a negative label for the training example;

determining an error of the neural network based on a semantic ranking loss of the label scores, wherein the semantic ranking loss is determined according to;

18. The system of claim 17, wherein D(y_c+j, y_c−

_k) is determined based on number of nodes traversed to travel between a leaf for the positive label y_c+j and a leaf for the negative label y<sub>c−

k</sub> in a semantic tree.

19. The system of claim 17, wherein determining the updated weights of the neural network comprises training the neural network with gradient descent backpropagation.