Image annotation based on label consensus

• US 10,013,436 B1
• Filed: 06/17/2015
• Issued: 07/03/2018
• Est. Priority Date: 06/17/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Cai, David, Zhou, Zhen Hao, Alldrin, Neil G., Duerig, Thomas J.
• Primary Examiner(s): Bibbee, Jared M
• Application Number:
  US14/741,833
• Time in Patent Office: 1,112 Days
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Implementations include actions of receiving an initial data set including a plurality of images, each image being associated with a set of labels, wherein each label in the set of labels is assigned to a respective image of the plurality of images by an initial model, the initial model being specific to a ground-truth label; for each image in the plurality of images: providing a list of categories associated with a respective image based on a respective set of labels, and determining a primary category of the respective image based on the list of categories; determining a category of the ground-truth label; and providing a revised data set based on the initial data set by comparing the category to primary categories of respective images in the plurality of images, the initial model being trained based on the revised data set to provide a revised model.

Claims

1. A computer-implemented method executed by one or more processors, the method comprising:

receiving, by the one or more processors, an initial data set comprising a plurality of images, each image from the plurality of images being associated with a set of labels, wherein each label in the set of labels is assigned to the image of the plurality of images by an initial model, the initial model being trained for a particular ground-truth label;

for each image in the plurality of images in the initial data set;

providing, by the one or more processors, a list of categories associated with the image based on the set of labels assigned to the image by the initial model, and

determining, by the one or more processors, a primary category of the image based on the list of categories;

determining, by the one or more processors, a category of the ground-truth label, the category having been specified for the ground-truth label of the initial model;

comparing, by the one or more processors, the category of the ground-truth label to primary categories of respective images in the plurality of images in the initial data set;

selecting, by the one or more processors, a revised data set, wherein the revised data set includes only images of the initial data set that are associated with a respective primary category that is the same as the category of the ground-truth label; and

providing, by the one or more processors, the revised data set to retrain the initial model to provide a revised model.

2. The method of claim 1, wherein providing a list of categories associated with the image based on the set of labels assigned to the image by the initial model comprises associating each label in the set of labels with a respective category based on a category map, the category map mapping entities to respective categories.

3. The method of claim 1, wherein determining a primary category of the image based on the list of categories comprises selecting a category having a highest category score in the list of categories as the primary category.

4. The method of claim 1, providing a list of categories associated with the image based on the set of labels comprises:

obtaining an expanded data set based on the initial data set;

providing, for each unique label in a set of unique labels, a label-specific histogram to provide a plurality of histograms;

providing, for each histogram of the plurality of histograms, a table to provide a plurality of tables, each table comprising one or more labels having respective label scores;

providing an aggregate table based on the plurality of tables, the aggregate table comprising an aggregate list of labels; and

assigning a category to each label in the aggregate list of labels to provide the list of categories.

5. The method of claim 4, wherein the expanded data set is provided based on processing training images associated with respective labels in the initial data set to provide an expanded set of labels, the set of unique labels being determined from the expanded set of labels.

6. The method of claim 1, wherein the revised model is used to label one or more received images.

7. A system comprising:

a data store for storing data; and

one or more processors configured to interact with the data store, the one or more processors being further configured to perform operations comprising;

receiving an initial data set comprising a plurality of images, each image from the plurality of images being associated with a set of labels, wherein each label in the set of labels is assigned to the image of the plurality of images by an initial model, the initial model being trained for a particular ground-truth label;

for each image in the plurality of images in the initial data set;

providing a list of categories associated with the image based on the set of labels assigned to the image by the initial model, and

determining a primary category of the respective image based on the list of categories;

determining a category of the ground-truth label, the category having been specified for the ground-truth label of the initial model;

comparing the category of the ground-truth label to primary categories of respective images in the plurality of images in the initial data set;

selecting a revised data set, wherein the revised data set includes only images of the initial data set that are associated with a respective primary category that is the same as the category of the ground-truth label; and

providing the revised data set to retrain the initial model to provide a revised model.

8. The system of claim 7, wherein providing a list of categories associated with the image based on the set of labels assigned to the image by the initial model comprises associating each label in the set of labels with a respective category based on a category map, the category map mapping entities to respective categories.

9. The system of claim 7, wherein determining a primary category of the respective image based on the list of categories comprises selecting a category having a highest category score in the list of categories as the primary category.

10. The system of claim 7, providing a list of categories associated with the image based on the set of labels comprises:

obtaining an expanded data set based on the initial data set;

providing, for each unique label in a set of unique labels, a label-specific histogram to provide a plurality of histograms;

providing, for each histogram of the plurality of histograms, a table to provide a plurality of tables, each table comprising one or more labels having respective label scores;

providing an aggregate table based on the plurality of tables, the aggregate table comprising an aggregate list of labels; and

assigning a category to each label in the aggregate list of labels to provide the list of categories.

11. The system of claim 10, wherein the expanded data set is provided based on processing training images associated with respective labels in the initial data set to provide an expanded set of labels, the set of unique labels being determined from the expanded set of labels.

12. The system of claim 7, wherein the revised model is used to label one or more received images.

13. A computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving an initial data set comprising a plurality of images, each image from the plurality of images being associated with a set of labels, wherein each label in the set of labels is assigned to the image of the plurality of images by an initial model, the initial model being trained for a particular ground-truth label in the initial data set;

for each image in the plurality of images;

providing a list of categories associated with the image based on the set of labels assigned to the image by the initial model, and

determining a primary category of the respective image based on the list of categories;

determining a category of the ground-truth label, the category having been specified for the ground-truth label of the initial model;

comparing the category of the ground-truth label to primary categories of respective images in the plurality of images in the initial data set;

selecting a revised data set, wherein the revised data set includes only images of the initial data set that are associated with a respective primary category that is the same as the category of the ground-truth label; and

providing the revised data set to retrain the initial model to provide a revised model.

14. The computer readable medium of claim 13, wherein providing a list of categories associated with the image based on the set of labels assigned by the initial model comprises associating each label in the set of labels with a respective category based on a category map, the category map mapping entities to respective categories.

15. The computer readable medium of claim 13, wherein determining a primary category of the image based on the list of categories comprises selecting a category having a highest category score in the list of categories as the primary category.

16. The computer readable medium of claim 13, providing a list of categories associated with the image based on the set of labels comprises:

obtaining an expanded data set based on the initial data set;

providing, for each unique label in a set of unique labels, a label-specific histogram to provide a plurality of histograms;

providing, for each histogram of the plurality of histograms, a table to provide a plurality of tables, each table comprising one or more labels having respective label scores;

providing an aggregate table based on the plurality of tables, the aggregate table comprising an aggregate list of labels; and

assigning a category to each label in the aggregate list of labels to provide the list of categories.

17. The computer readable medium of claim 16, wherein the expanded data set is provided based on processing training images associated with respective labels in the initial data set to provide an expanded set of labels, the set of unique labels being determined from the expanded set of labels.

18. The computer readable medium of claim 13, wherein the revised model is used to label one or more received images.