Image annotation based on label consensus

• US 10,185,725 B1
• Filed: 05/31/2018
• Issued: 01/22/2019
• Est. Priority Date: 06/17/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Cai, David, Zhou, Zhen Hao, Alldrin, Neil G., Duerig, Thomas J.
• Primary Examiner(s): Bibbee, Jared M
• Application Number:
  US15/993,833
• Time in Patent Office: 236 Days
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Implementations include actions of receiving an initial data set including a plurality of images, each image being associated with a set of labels, wherein each label in the set of labels is assigned to a respective image of the plurality of images by an initial model, the initial model being specific to a ground-truth label; for each image in the plurality of images: providing a list of categories associated with a respective image based on a respective set of labels, and determining a primary category of the respective image based on the list of categories; determining a category of the ground-truth label; and providing a revised data set based on the initial data set by comparing the category to primary categories of respective images in the plurality of images, the initial model being trained based on the revised data set to provide a revised model.

Claims

1. A computer-implemented method executed by one or more processors, the method comprising:

receiving, by the one or more processors, an initial data set comprising a plurality of images, each image from the plurality of images being associated with a set of labels, wherein each label in the set of labels is assigned to the image of the plurality of images by an initial model, the initial model being trained for a particular ground-truth label;

for each image in the plurality of images in the initial data set, predicting, using the initial model, a set of top k predicted labels for the initial data set;

determining, from the sets of top k predicted labels, a set of unique labels;

for each unique label in the set of unique labels;

selecting a respective set of training images for the unique label;

predicting, using the initial model, a set of top k predicted labels for the respective set of training images; and

generating, for each label of the top k predicted labels, a value based on the number of times the respective label occurs in the top k predicted labels in respective set of training images for the unique label;

generating, from the values of each of the labels in the top k predicted labels, a mapping of the unique label to labels in the top k predicted labels that indicates relative strengths of the unique label to the labels in the top k predicted labels;

determining, from the mapping, categories for each unique label; and

determining, for each image in the initial data set, a primary category of the image based on the list of categories of the set of labels for the image.

2. The computer-implemented method of claim 1, further comprising:

comparing, by the one or more processors, the category of the ground-truth label to primary categories of respective images in the plurality of images in the initial data set;

selecting, by the one or more processors, a revised data set, wherein the revised data set includes only images of the initial data set that are associated with a respective primary category that is the same as the category of the ground-truth label; and

providing, by the one or more processors, the revised data set to retrain the initial model to provide a revised model.

3. The method of claim 1, wherein determining, from the mapping, categories for each unique label comprises associating each label in the top k predicted labels with a respective category based on a category map, the category map mapping entities to respective categories.

4. The method of claim 1, wherein determining a primary category of the image based on the list of categories comprises selecting a category having a highest category score in the list of categories as the primary category.

5. The method of claim 1, wherein the expanded data set is provided based on processing training images associated with respective labels in the initial data set to provide an expanded set of labels, the set of unique labels being determined from the expanded set of labels.

6. The method of claim 1, wherein the revised model is used to label one or more received images.

7. A system comprising:

a data store for storing data; and

one or more processors configured to interact with the data store, the one or more processors being further configured to perform operations comprising;

receiving, by the one or more processors, an initial data set comprising a plurality of images, each image from the plurality of images being associated with a set of labels, wherein each label in the set of labels is assigned to the image of the plurality of images by an initial model, the initial model being trained for a particular ground-truth label;

for each image in the plurality of images in the initial data set, predicting, using the initial model, a set of top k predicted labels for the initial data set;

determining, from the sets of top k predicted labels, a set of unique labels;

for each unique label in the set of unique labels;

selecting a respective set of training images for the unique label;

predicting, using the initial model, a set of top k predicted labels for the respective set of training images; and

generating, for each label of the top k predicted labels, a value based on the number of times the respective label occurs in the top k predicted labels in respective set of training images for the unique label;

generating, from the values of each of the labels in the top k predicted labels, a mapping of the unique label to labels in the top k predicted labels

determining, from the mapping, categories for each unique label; and

determining, for each image in the initial data set, a primary category of the image based on the list of categories of the set of labels for the image.

8. The system of claim 7, the operations further comprising:

comparing, by the one or more processors, the category of the ground-truth label to primary categories of respective images in the plurality of images in the initial data set;

selecting, by the one or more processors, a revised data set, wherein the revised data set includes only images of the initial data set that are associated with a respective primary category that is the same as the category of the ground-truth label; and

providing, by the one or more processors, the revised data set to retrain the initial model to provide a revised model.

9. The system of claim 7, wherein determining, from the mapping, categories for each unique label comprises associating each label in the top k predicted labels with a respective category based on a category map, the category map mapping entities to respective categories.

10. The system of claim 7, wherein determining a primary category of the image based on the list of categories comprises selecting a category having a highest category score in the list of categories as the primary category.

11. The system of claim 7, wherein the expanded data set is provided based on processing training images associated with respective labels in the initial data set to provide an expanded set of labels, the set of unique labels being determined from the expanded set of labels.

12. The system of claim 7, wherein the revised model is used to label one or more received images.

13. A computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

receiving, by the one or more processors, an initial data set comprising a plurality of images, each image from the plurality of images being associated with a set of labels, wherein each label in the set of labels is assigned to the image of the plurality of images by an initial model, the initial model being trained for a particular ground-truth label;

for each image in the plurality of images in the initial data set, predicting, using the initial model, a set of top k predicted labels for the initial data set;

determining, from the sets of top k predicted labels, a set of unique labels;

for each unique label in the set of unique labels;

selecting a respective set of training images for the unique label;

predicting, using the initial model, a set of top k predicted labels for the respective set of training images; and

generating, for each label of the top k predicted labels, a value based on the number of times the respective label occurs in the top k predicted labels in respective set of training images for the unique label;

generating, from the values of each of the labels in the top k predicted labels, a mapping of the unique label to labels in the top k predicted labels

determining, from the mapping, categories for each unique label; and

determining, for each image in the initial data set, a primary category of the image based on the list of categories of the set of labels for the image.

14. The computer-readable medium of claim 13, the operations further comprising:

comparing, by the one or more processors, the category of the ground-truth label to primary categories of respective images in the plurality of images in the initial data set;

selecting, by the one or more processors, a revised data set, wherein the revised data set includes only images of the initial data set that are associated with a respective primary category that is the same as the category of the ground-truth label; and

providing, by the one or more processors, the revised data set to retrain the initial model to provide a revised model.

15. The computer-readable medium of claim 13, wherein determining, from the mapping, categories for each unique label comprises associating each label in the top k predicted labels with a respective category based on a category map, the category map mapping entities to respective categories.

16. The computer-readable medium of claim 13, wherein determining a primary category of the image based on the list of categories comprises selecting a category having a highest category score in the list of categories as the primary category.

17. The computer-readable medium of claim 13, wherein the expanded data set is provided based on processing training images associated with respective labels in the initial data set to provide an expanded set of labels, the set of unique labels being determined from the expanded set of labels.

18. The computer-readable medium of claim 13, wherein the revised model is used to label one or more received images.