System and method for generating a classifier for semantically segmenting an image

• US 8,958,630 B1
• Filed: 11/02/2011
• Issued: 02/17/2015
• Est. Priority Date: 10/24/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Gallup, David, Raguram, Rahul
• Primary Examiner(s): CONWAY, THOMAS A
• Application Number:
  US13/287,890
• Time in Patent Office: 1,203 Days
• Field of Search: None
• US Class Current: 382/159
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06V 10/7753 Incorporation of unlabelled data, e.g. multiple instance learning [MIL]
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Systems, methods, and machine-readable media for generating a classifier configured to label segments of an image, are discussed. According to one aspect, the system may include a training module, a labeling module, and an update module. The training module may be configured to train a first sub-classifier based on photographic data for a set of pre-labeled image segments and a second sub-classifier based on 3-dimensional point data for the set of pre-labeled image segments. The labeling module may be configured to generate a labeling solution comprising a plurality of associations between an image segment from the set of unlabeled image segments and a label. The update module may be configured to update the set of pre-labeled image segments based on the labeling solution. The training module may also be configured to train the first sub-classifier and the second sub-classifier based on the updated set of pre-labeled image segments.

Claims

1. A method for generating a classifier configured to automatically label segments of an image, wherein the classifier comprises a first sub-classifier and a second sub-classifier, the method comprising:

training a first sub-classifier based on photographic data for a labeled set of image segments and a second sub-classifier based on 3-dimensional (3D) point data for the labeled set of image segments, wherein each of the labeled image segments can be a portion of a larger image partitioned into a plurality of the image segments based on similarity of pixels within the segment and differences with pixels which are outside a boundary of the segment;

automatically creating, based on the training, a labeling solution for an unlabeled, second set of image segments by running the first sub-classifier on the second set of image segments and running the second sub-classifier on the second set of image segments, wherein the labeling solution comprises a plurality of associations, each association of the plurality of associations linking an image segment from the set of unlabeled image segments with a label;

updating the labeled set of image segments based on the labeling solution, including adding an image segment of the second segments to the labeled set of image segments together with a label having at least one of the associations with the added segment; and

retraining the first sub-classifier and the second sub-classifier based on the updated labeled set of image segments.

2. The method of claim 1, wherein the creating, the updating, and the retraining is repeated for at least one iteration.

3. The method of claim 1, wherein the running of the first sub-classifier on the second set image segments is based on photographic data for the second set of image segments and wherein the running of the second sub-classifier on the second set of image segments is based on 3D point data for the second set of image segments.

4. The method of claim 1, wherein the training of the first sub-classifier based on photographic data for the labeled set of image segments comprises:

for each image segment in the labeled set of image segments;

identifying a label corresponding to the image segment,

determining photographic data features for the image segment, and

associating the photographic data features with the identified label; and

wherein the training of the second sub-classifier based on 3-dimensional (3D) point data for the labeled set of image segments comprises, for each image segment in the first set of image segments;

identifying the label corresponding to the image segment,

determining 3D point data features for the image segment, and

associating the 3D point data features with the identified label.

5. The method of claim 1, further comprising:

receiving the labeled set of image segments, wherein each image segment in the labeled set of image segments is associated with photographic data and 3-dimensional (3D) point data for the image segment; and

receiving the second set of image segments, wherein each image segment in the second set of image segments is associated with photographic data and 3-dimensional (3D) point data for the unlabeled image segment.

6. The method of claim 1, wherein each association of the labeling solution is associated with a confidence value.

7. The method of claim 6, wherein the updating further comprises:

selecting the label from the labeling solution based on the confidence value associated with the selected association; and

removing the image segment from the second set of image segments,

wherein the adding adds the image segment and the label for the selected association to the labeled set of image segments.

8. A system for generating a classifier configured to label segments of an image, wherein the classifier comprises a first sub-classifier and a second sub-classifier, the system comprising:

at least one processor; and

a plurality of modules, each module including at least one of;

a hardware component, or a plurality of instructions executable by the at least one processor, the modules including;

a training module configured to train a first sub-classifier based on photographic data for a set of pre-labeled image segments and a second sub-classifier based on 3-dimensional (3D) point data for the set of pre-labeled image segments, wherein each of the pre-labeled image segments can be a portion of a larger image partitioned into a plurality of the pre-labeled image segments based on similarity of pixels within the segment and differences with pixels which are outside a boundary of the segment;

a labeling module configured to generate a labeling solution comprising a plurality of associations between each of a plurality of image segments from a set of unlabeled image segments and a corresponding label of a plurality of labels, wherein the labeling solution is generated by running the first sub-classifier on the set of unlabeled image segments and running the second sub-classifier on the set of unlabeled image segments; and

an update module configured to update the set of pre-labeled image segments based on the labeling solution generated by the first sub-classifier and the second sub-classifier wherein the update includes adding an image segment of the unlabeled image segments to the pre-labeled image segments together with a label having at least one of the associations with the added segment

wherein the training module is further configured to retrain the first sub-classifier and the second sub-classifier based on the updated set of pre-labeled image segments.

9. The system of claim 8, wherein the labeling module, the update module, and the training module are configured to generate the labeling solution, update the set of pre-labeled image segments, and retrain the first sub-classifier and the second sub-classifier based on the updated set of pre-labeled image segments for more than one iteration.

10. The system of claim 8, wherein the training module is configured to train of the first sub-classifier by:

for each pre-labeled image segment in the set of pre-labeled image segments;

identifying a label corresponding to the pre-labeled image segment,

determining photographic data features for the pre-labeled image segment, and

associating the photographic data features with the identified label; and

wherein the training module is configured to train of the second sub-classifier by, for each pre-labeled image segment in the set of pre-labeled image segments;

identifying the label corresponding to the pre-labeled image segment,

determining 3D point data features for the pre-labeled image segment, and

associating the 3D point data features with the identified label.

11. The system of claim 8, wherein each association in the labeling solution has a corresponding a confidence value.

12. The system of claim 11, wherein the update module is configured to update the set of pre-labeled image segments by:

selecting an association of an image segment from the set of unlabeled image segments and a label from the labeling solution based on the confidence value associated with the selected association;

removing the image segment from the set of unlabeled image segments; and

wherein the adding adds the image segment and the label for the selected association to the set of pre-labeled image segments.

13. The system of claim 8, wherein the plurality of modules further comprises an interface module configured to:

receive the set of pre-labeled image segments, wherein each pre-labeled image segment is associated with photographic data and 3-dimensional (3D) point data for the pre-labeled image segment; and

receive the set of unlabeled image segments, wherein each unlabeled image segment is associated with photographic data and 3-dimensional (3D) point data for the unlabeled image segment.

14. The system of claim 13, wherein the set of pre-labeled image segments received by the interface module are manually labeled.

15. A non-transitory machine-readable medium comprising instructions stored therein, which when executed by a machine, cause the machine to perform operations comprising:

training a first sub-classifier based on photographic data for a set of pre-labeled image segments and a second sub-classifier based on 3-dimensional (3D) point data for the set of pre-labeled image segments, wherein each of the pre-labeled image segments can be a portion of a larger image partitioned into a plurality of the segments based on similarity of pixels within the segment and differences which are outside a boundary of the segment; and

for at least one iteration;

generating a labeling solution by running the first sub-classifier on a set of unlabeled image segments and running the second sub-classifier on the set of unlabeled image segments,

wherein the running of the first sub-classifier on the set of unlabeled image segments is based on photographic data for the set of unlabeled image segments and wherein the running of the second sub-classifier on the set of unlabeled image segments is based on 3D point data for the set of unlabeled image segments,

and wherein the labeling solution comprises a plurality of associations, each association of the plurality of associations linking an image segment from the set of unlabeled image segments with a label, and wherein each association is associated with a confidence value,

updating the set of pre-labeled image segments based on the labeling solution generated by the first sub-classifier and the second sub-classifier, including adding an image segment of the unlabeled image segments to the pre-labeled image segments together with a label having at least one of the associations with the added segment, and

training the first sub-classifier and the second sub-classifier based on the updated set of pre-labeled image segments.

16. The non-transitory machine-readable medium of claim 15, wherein the updating comprises:

selecting an association of an image segment from the set of unlabeled image segments and a label from the labeling solution based on the confidence value associated with the selected association;

removing the image segment from the set of unlabeled image segments; and

wherein the adding adds the image segment and the label for the selected association to the set of pre-labeled image segments.

17. The non-transitory machine-readable medium of claim 15, wherein the training of the first sub-classifier based on photographic data for the set of pre-labeled image segments comprises:

for each pre-labeled image segment in the set of pre-labeled image segments;

identifying a label corresponding to the pre-labeled image segment,

determining photographic data features for the pre-labeled image segment, and

associating the photographic data features with the identified label; and

wherein the training of the second sub-classifier based on 3-dimensional (3D) point data for the set of pre-labeled image segments comprises, for each pre-labeled image segment in the set of pre-labeled image segments;

identifying the label corresponding to the pre-labeled image segment,

determining 3D point data features for the pre-labeled image segment, and

associating the 3D point data features with the identified label.