Methods, systems, and programming for annotating an image including scoring using a plurality of trained classifiers corresponding to a plurality of clustered image groups associated with a set of weighted labels

• US 8,909,563 B1
• Filed: 06/17/2011
• Issued: 12/09/2014
• Est. Priority Date: 06/17/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Jing, Yushi, Liu, Yi, Tsai, David
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Brown, Jr., Nathan
• Application Number:
  US13/163,317
• Time in Patent Office: 1,271 Days
• Field of Search: 706/12
• US Class Current: 706/12
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/232 Non-hierarchical techniques
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

Methods, systems, and articles of manufacture for annotating of an image are disclosed. These include scoring the image using a plurality of trained classifiers, wherein each of the trained classifiers corresponds to at least one of a plurality of image groups clustered based upon image similarity, and wherein each image group is associated with a set of weighted labels; selecting one or more of the image groups based upon the scoring; aggregating one or more sets of weighted labels associated with the selected one or more image groups; and annotating the image using the aggregated one or more sets of weighted labels.

Claims

1. A method of annotating an image, comprising:

scoring the image using a plurality of trained classifiers, wherein each of the trained classifiers corresponds to at least one of a plurality of image groups clustered based upon image similarity, and wherein each image group is associated with a set of weighted labels,

wherein each of the weighted labels includes a label and an associated weight, and wherein the associated weight is determined based upon the frequency of occurrence of the label in the image group;

selecting one or more of the image groups based upon the scoring;

aggregating one or more sets of weighted labels associated with the selected one or more image groups; and

annotating the image using the aggregated one or more sets of weighted labels.

2. A method of annotating an image, comprising:

scoring the image using a plurality of trained classifiers, wherein each of the trained classifiers corresponds to at least one of a plurality of image groups clustered based upon image similarity, and wherein each image group is associated with a set of weighted labels;

selecting one or more of the image groups based upon the scoring;

aggregating one or more sets of weighted labels associated with the selected one or more image groups;

training a plurality of classifiers to create the plurality of trained classifiers, wherein at least one of the classifiers is trained for each of the respective image groups; and

annotating the image using the aggregated one or more sets of weighted labels.

3. The method of claim 2, wherein each of the plurality of classifiers is trained using a linear model.

4. The method of claim 2, wherein a classifier from the plurality of classifiers is trained for a selected one of the image groups using a positive training sample and a negative training sample, wherein the positive training sample comprises images from the selected group, and wherein the negative training sample comprises images from neighboring groups of the selected group.

5. The method of claim 4, wherein the negative training sample comprises images from neighboring groups of the selected group and images from all other groups of the image groups.

6. A method of annotating an image, comprising:

scoring the image using a plurality of trained classifiers, wherein each of the trained classifiers corresponds to at least one of a plurality of image groups clustered based upon image similarity, and wherein each image group is associated with a set of weighted labels,

wherein the scoring the image comprises assigning, corresponding to respective ones of the image groups, a score to the image based upon a distance of a set of features of the image to a set of features representative of the image group;

selecting one or more of the image groups based upon the scoring;

aggregating one or more sets of weighted labels associated with the selected one or more image groups; and

annotating the image using the aggregated one or more sets of weighted labels.

7. A method of annotating an image, comprising:

scoring the image using a plurality of trained classifiers, wherein each of the trained classifiers corresponds to at least one of a plurality of image groups clustered based upon image similarity, and wherein each image group is associated with a set of weighted labels;

selecting one or more of the image groups based upon the scoring, wherein the selecting one or more of the groups comprises filtering ones of the plurality of image groups that have a score of less than a threshold;

aggregating one or more sets of weighted labels associated with the selected one or more image groups; and

annotating the image using the aggregated one or more sets of weighted labels.

8. A system for annotating an image, comprising:

a processor;

a memory coupled to the processor;

a classification and scoring module configured to execute on the processor and further configured to;

score an image using a plurality of trained classifiers, wherein each of the trained classifiers corresponds to at least one of a plurality of image groups clustered based upon image similarity, and wherein each image group is associated with a set of weighted labels,

wherein each of the weighted labels includes a label and an associated weight, and wherein the associated weight is determined based upon the frequency of occurrence of the label in the image group; and

select one or more of the image groups based upon the scoring; and

a label aggregator module configured to execute on the processor and further configured to;

aggregate one or more sets of weighted labels associated with the selected one or more image groups; and

annotate the image using the aggregated one or more sets of weighted labels.

9. A system for annotating an image, comprising:

a processor;

a memory coupled to the processor;

a classification and scoring module configured to execute on the processor and further configured to;

score an image using a plurality of trained classifiers, wherein each of the trained classifiers corresponds to at least one of a plurality of image groups clustered based upon image similarity, and wherein each image group is associated with a set of weighted labels;

select one or more of the image groups based upon the scoring,

wherein the classification and scoring module is further configured to train a plurality of classifiers to create the plurality of trained classifiers, wherein at least one of the classifiers is trained for each of the respective image groups; and

a label aggregator module configured to execute on the processor and further configured to;

aggregate one or more sets of weighted labels associated with the selected one or more image groups; and

annotate the image using the aggregated one or more sets of weighted labels.