Rare instance classifiers

• US 10,346,724 B2
• Filed: 06/22/2017
• Issued: 07/09/2019
• Est. Priority Date: 06/22/2017
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Lo, Wan-Yen, Ogale, Abhijit, Gao, Yang
• Primary Examiner(s): Summers, Geoffrey E
• Application Number:
  US15/630,275
• Publication Number:
  US 20180373963A1
• Time in Patent Office: 747 Days
• Field of Search: 382156, 382157, 382158, 382104
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/582 of traffic signs

Abstract

In some implementations, an image classification system of an autonomous or semi-autonomous vehicle is capable of improving multi-object classification by reducing repeated incorrect classification of objects that are considered rarely occurring objects. The system can include a common instance classifier that is trained to identify and recognize general objects (e.g., commonly occurring objects and rarely occurring objects) as belonging to specified object categories, and a rare instance classifier that is trained to compute one or more rarity scores representing likelihoods that an input image is correctly classified by the common instance classifier. The output of the rare instance classifier can be used to adjust the classification output of the common instance classifier such that the likelihood of input images being incorrectly classified is reduced.

Claims

1. A method, comprising:

receiving an input image;

processing the input image using a common instance neural network, wherein the common instance neural network is configured to process the input image to generate a common instance output comprising a respective first object score corresponding to each of one or more first object categories, each first object score representing a likelihood that the input image includes an image of an object belonging to the corresponding first object category;

processing the input image using a rare instance neural network, wherein the rare instance neural network is configured to process the input image to generate a rare instance output comprising a rarity score that represents a likelihood that the input image will be incorrectly classified by the common instance neural network;

determining a weight to be assigned to the one or more respective first object scores in classifying the input image using the rarity score; and

classifying the input image in accordance with the determined weight.

2. The method of claim 1, wherein the rare instance neural network has fewer parameters than the common instance neural network.

3. The method of claim 1, wherein the rare instance neural network has been trained on training images that were misclassified by the common instance neural network.

4. The method of claim 1, wherein the rare instance neural network has been trained on images of object types that occur rarely in images used to train the common instance neural network.

5. The method of claim 1, wherein:

the rare instance output further comprises a respective second object score corresponding to each of one or more second object categories, each second object score representing a likelihood that the input image includes an image of an object belonging to the corresponding second object category; and

the method further comprises determining a weight to assign the one or more second object scores in classifying the input image using the rarity score.

6. The method of claim 5, wherein the second object categories include one or more of the first object categories.

7. The method of claim 5, wherein determining a weight to assign the one or more second object scores comprise using the rarity score as the weight for the second object scores.

8. The method of claim 5, wherein determining a weight to assign the one or more second object scores and the one or more first object scores comprises:

determining that the rarity score does not satisfy a predetermined threshold, and

in response, reducing the weight assigned to the one or more respective second object scores in classifying the input image.

9. The method of claim 5, wherein determining a weight to assign the one or more second object scores and the one or more first object scores comprises:

determining that the rarity score satisfies a predetermined threshold, and

in response determining that the rarity score satisfies a predetermined threshold, reducing the weight assigned to the one or more respective first object scores in classifying the input image.

10. The method of claim 9, wherein reducing the weight assigned to the one or more respective first object scores comprise assigning a weight to a value that indicates that the first object scores are not used in classifying the input image.

11. The method of claim 1, wherein classifying the input image comprises:

processing the common instance output and the rare instance output using a combining neural network, wherein the combing neural network configured to compute a respective third object score corresponding to each of the one or more first object categories, each third object score computed based on the determined weights assigned to the one or more respective first object scores and the determined weights assigned to the one or more respective second object scores; and

classifying the input image based on the respective values of the third object scores.

12. The method of claim 1, wherein determining the weight to assign the one or more respective first object scores comprise:

determining that the rarity score satisfies a predetermined threshold; and

in response to determining that a rarity score satisfies a predetermined threshold, reducing the weight assigned to the one or more respective first object scores in classifying the input image.

13. A system comprising:

one or more computers; and

one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

receiving an input image;

processing the input image using a common instance neural network, wherein the common instance neural network is configured to process the input image to generate a common instance output comprising a respective first object score corresponding to each of one or more first object categories, each first object score representing a likelihood that the input image includes an image of an object belonging to the corresponding first object category;

processing the input image using a rare instance neural network, wherein the rare instance neural network is configured to process the input image to generate a rare instance output comprising a rarity score that represents a likelihood that the input image will be incorrectly classified by the common instance neural network;

determining a weight to be assigned to the one or more respective first object scores in classifying the input image using the rarity score; and

classifying the input image in accordance with the determined weight.

14. The system of claim 13, wherein the rare instance neural network has fewer parameters than the common instance neural network.

15. The system of claim 13, wherein the rare instance neural network has been trained on training images that were misclassified by the common instance neural network.

16. A non-transitory computer-readable storage device encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

receiving an input image;

processing the input image using a common instance neural network, wherein the common instance neural network is configured to process the input image to generate a common instance output comprising a respective first object score corresponding to each of one or more first object categories, each first object score representing a likelihood that the input image includes an image of an object belonging to the corresponding first object category;

processing the input image using a rare instance neural network, wherein the rare instance neural network is configured to process the input image to generate a rare instance output comprising a rarity score that represents a likelihood that the input image will be incorrectly classified by the common instance neural network;

determining a weight to be assigned to the one or more respective first object scores in classifying the input image using the rarity score; and

classifying the input image in accordance with the determined weight.

17. The device of claim 16, wherein the rare instance neural network has fewer parameters than the common instance neural network.

18. The device of claim 16, wherein the rare instance neural network has been trained on training images that were misclassified by the common instance neural network.

19. The device of claim 16, wherein the rare instance neural network has been trained on images of object types that occur rarely in images used to train the common instance neural network.

20. The device of claim 16, wherein:

the rare instance output further comprises a respective second object score corresponding to each of one or more second object categories, each second object score representing a likelihood that the input image includes an image of an object belonging to the corresponding second object category; and

the method further comprises determining a weight to assign the one or more second object scores in classifying the input image using the rarity score.