CATEGORY LEARNING NEURAL NETWORKS

• US 20200027002A1
• Filed: 07/15/2019
• Published: 01/23/2020
• Est. Priority Date: 07/20/2018
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Hickson, Steven, Angelova, Anelia, Essa, Irfan Aziz, Sukthankar, Rahul
• Granted Patent: US 10,635,979 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/232 Non-hierarchical techniques
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining a clustering of images into a plurality of semantic categories. In one aspect, a method comprises: training a categorization neural network, comprising, at each of a plurality of iterations: processing an image depicting an object using the categorization neural network to generate (i) a current prediction for whether the image depicts an object or a background region, and (ii) a current embedding of the image; determining a plurality of current cluster centers based on the current values of the categorization neural network parameters, wherein each cluster center represents a respective semantic category; and determining a gradient of an objective function that includes a classification loss and a clustering loss, wherein the clustering loss depends on a similarity between the current embedding of the image and the current cluster centers.

Claims

1. A method, comprising:

training a categorization neural network to determine trained values of the categorization neural network parameters from initial values of the categorization neural network parameters, comprising, at each of a plurality of iterations;

processing an image depicting an object using the categorization neural network in accordance with current values of categorization neural network parameters to generate an output comprising;

(i) a current prediction for whether the image depicts an object or a background region, and (ii) a current embedding of the image;

determining a plurality of current cluster centers based on the current values of the categorization neural network parameters, wherein each cluster center represents a respective semantic category;

determining a gradient of an objective function that includes a classification loss and a clustering loss, wherein the classification loss depends on the current prediction for whether the image depicts an object or a background region, and wherein the clustering loss depends on a similarity between the current embedding of the image and the current cluster centers; and

determining an update to the current values of the categorization neural network parameters from the gradient;

determining a plurality of final cluster centers based on the trained values of the categorization neural network parameters;

for each of a plurality of images, processing the image using the categorization neural network in accordance with the trained values of the categorization neural network parameters to generate a final embedding of the image; and

determining a clustering of the images into a plurality of semantic categories using the final embeddings of the images and the final cluster centers.

2. The method of claim 1, wherein the images are generated by a plurality of operations comprising, for each of a plurality of training images:

generating a depth-augmented training image by determining a depth associated with each pixel in the training image;

clustering the pixels of the depth-augmented training image using;

(i) the intensity data associated with the pixels of the training image, and (ii) the depths of the pixels of the training image; and

generating one or more images depicting objects based on the clustering of the pixels of the depth-augmented training image.

3. The method of claim 2, wherein determining a depth associated with each pixel in the training image comprises:

processing the training image by using a depth estimation neural network in accordance with trained values of depth estimation neural network parameters to generate an output comprising a depth associated with each pixel in the training image.

4. The method of claim 3, wherein the depth estimation neural network is trained using an unsupervised machine learning training technique.

5. The method of claim 2, wherein generating one or more images depicting objects using the clustering of the pixels of the depth-augmented training image comprises:

cropping the images depicting objects from the training image based on the clustering of the pixels of the depth-augmented training image.

6. The method of claim 1, wherein determining the current cluster centers based on the current values of the categorization neural network parameters comprises:

obtaining the current cluster centers from a memory unit of the categorization neural network.

7. The method of claim 1, wherein the clustering loss comprises:

a minimum over each current cluster center of a difference between the current cluster center and the current embedding of the image.

8. The method of claim 1, wherein the clustering loss further comprises:

a measure of how evenly images are distributed between the current cluster centers.

9. The method of claim 1, wherein determining the clustering of the images into the plurality of semantic categories using the final embeddings of the images and the final cluster centers comprises:

for each image, assigning the image to a closest final cluster center to the final embedding of the image; and

for each final cluster center, determining the images assigned to the final cluster center as belonging to a same semantic category.

10. The method of claim 1, wherein the categorization neural network comprises a plurality of convolutional neural network layers.

11. The method of claim 1, wherein the current embedding of the image is an intermediate output of the categorization neural network.

12. The method of claim 1 further comprising, at each of a plurality of training iterations:

processing an image depicting a background region using the categorization neural network in accordance with current values of categorization neural network parameters to generate an output comprising a current prediction for whether the image depicts an object or a background region;

determining a gradient of an objective function that includes the classification loss; and

determining an update to the current values of the categorization neural network parameters from the gradient.

13. A system, comprising:

a data processing apparatus;

a memory in data communication with the data processing apparatus and storing instructions that cause the data processing apparatus to perform operations comprising;

training a categorization neural network to determine trained values of the categorization neural network parameters from initial values of the categorization neural network parameters, comprising, at each of a plurality of iterations;

processing an image depicting an object using the categorization neural network in accordance with current values of categorization neural network parameters to generate an output comprising;

(i) a current prediction for whether the image depicts an object or a background region, and (ii) a current embedding of the image;

determining a plurality of current cluster centers based on the current values of the categorization neural network parameters, wherein each cluster center represents a respective semantic category;

determining a gradient of an objective function that includes a classification loss and a clustering loss, wherein the classification loss depends on the current prediction for whether the image depicts an object or a background region, and wherein the clustering loss depends on a similarity between the current embedding of the image and the current cluster centers; and

determining an update to the current values of the categorization neural network parameters from the gradient;

determining a plurality of final cluster centers based on the trained values of the categorization neural network parameters;

for each of a plurality of images, processing the image using the categorization neural network in accordance with the trained values of the categorization neural network parameters to generate a final embedding of the image; and

determining a clustering of the images into the plurality of semantic categories using the final embeddings of the images and the final cluster centers.

14. The system of claim 13, wherein the images are generated by a plurality of operations comprising, for each of a plurality of training images:

generating a depth-augmented training image by determining a depth associated with each pixel in the training image;

clustering the pixels of the depth-augmented training image using;

(i) the intensity data associated with the pixels of the training image, and (ii) the depths of the pixels of the training image; and

generating one or more images depicting objects based on the clustering of the pixels of the depth-augmented training image.

15. The system of claim 14, wherein determining a depth associated with each pixel in the training image comprises:

processing the training image by using a depth estimation neural network in accordance with trained values of depth estimation neural network parameters to generate an output comprising a depth associated with each pixel in the training image.

16. The method of claim 15, wherein the depth estimation neural network is trained using an unsupervised machine learning training technique.

17. One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

training a categorization neural network to determine trained values of the categorization neural network parameters from initial values of the categorization neural network parameters, comprising, at each of a plurality of iterations;

processing an image depicting an object using the categorization neural network in accordance with current values of categorization neural network parameters to generate an output comprising;

(i) a current prediction for whether the image depicts an object or a background region, and (ii) a current embedding of the image;

determining a plurality of current cluster centers based on the current values of the categorization neural network parameters, wherein each cluster center represents a respective semantic category;

determining a gradient of an objective function that includes a classification loss and a clustering loss, wherein the classification loss depends on the current prediction for whether the image depicts an object or a background region, and wherein the clustering loss depends on a similarity between the current embedding of the image and the current cluster centers; and

determining an update to the current values of the categorization neural network parameters from the gradient;

determining a plurality of final cluster centers based on the trained values of the categorization neural network parameters;

for each of a plurality of images, processing the image using the categorization neural network in accordance with the trained values of the categorization neural network parameters to generate a final embedding of the image; and

determining a clustering of the images into the plurality of semantic categories using the final embeddings of the images and the final cluster centers.

18. The computer storage media of claim 17, wherein the images are generated by a plurality of operations comprising, for each of a plurality of training images:

generating a depth-augmented training image by determining a depth associated with each pixel in the training image;

clustering the pixels of the depth-augmented training image using;

(i) the intensity data associated with the pixels of the training image, and (ii) the depths of the pixels of the training image; and

generating one or more images depicting objects based on the clustering of the pixels of the depth-augmented training image.

19. The computer storage media of claim 18, wherein determining a depth associated with each pixel in the training image comprises:

processing the training image by using a depth estimation neural network in accordance with trained values of depth estimation neural network parameters to generate an output comprising a depth associated with each pixel in the training image.

20. The computer storage media of claim 19, wherein the depth estimation neural network is trained using an unsupervised machine learning training technique.