Category learning neural networks

• US 10,635,979 B2
• Filed: 07/15/2019
• Issued: 04/28/2020
• Est. Priority Date: 07/20/2018
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Hickson, Steven, Angelova, Anelia, Essa, Irfan Aziz, Sukthankar, Rahul
• Primary Examiner(s): Chen, Xuemei G
• Application Number:
  US16/511,637
• Publication Number:
  US 20200027002A1
• Time in Patent Office: 288 Days
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/232 Non-hierarchical techniques
• CPC Class Codes: G06F 18/24137 Distances to cluster centroïds
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining a clustering of images into a plurality of semantic categories. In one aspect, a method comprises: training a categorization neural network, comprising, at each of a plurality of iterations: processing an image depicting an object using the categorization neural network to generate (i) a current prediction for whether the image depicts an object or a background region, and (ii) a current embedding of the image; determining a plurality of current cluster centers based on the current values of the categorization neural network parameters, wherein each cluster center represents a respective semantic category; and determining a gradient of an objective function that includes a classification loss and a clustering loss, wherein the clustering loss depends on a similarity between the current embedding of the image and the current cluster centers.

Claims

1. A method, comprising:

training a categorization neural network to determine trained values of the categorization neural network parameters from initial values of the categorization neural network parameters, comprising, at each of a plurality of iterations;

processing a given image depicting an object using the categorization neural network in accordance with current values of categorization neural network parameters to generate an output comprising;

(i) a current prediction for whether the given image depicts an object or a background region, and (ii) a current embedding of the given image;

determining a plurality of current cluster centers based on the current values of the categorization neural network parameters, wherein each cluster center represents a respective semantic category;

determining a gradient of an objective function that includes a classification loss and a clustering loss, wherein the classification loss depends on the current prediction for whether the given image depicts an object or a background region, and wherein the clustering loss depends on a similarity between the current embedding of the given image and the current cluster centers; and

determining an update to the current values of the categorization neural network parameters from the gradient;

determining a plurality of final cluster centers based on the trained values of the categorization neural network parameters;

for each of a plurality of target images, processing the target image using the categorization neural network in accordance with the trained values of the categorization neural network parameters to generate a final embedding of the target image; and

determining a clustering of the target images into a plurality of semantic categories using the final embeddings of the target images and the final cluster centers.

2. The method of claim 1, wherein the given image is generated by a plurality of operations comprising:

generating a depth-augmented training image by determining a depth associated with each pixel in a training image;

clustering the pixels of the depth-augmented training image using;

(i) the intensity data associated with the pixels of the training image, and (ii) the depths of the pixels of the training image; and

generating the given image based on the clustering of the pixels of the depth-augmented training image.

3. The method of claim 2, wherein determining a depth associated with each pixel in the training image comprises:

processing the training image by using a depth estimation neural network in accordance with trained values of depth estimation neural network parameters to generate an output comprising a depth associated with each pixel in the training image.

4. The method of claim 3, wherein the depth estimation neural network is trained using an unsupervised machine learning training technique.

5. The method of claim 2, wherein generating the given image using the clustering of the pixels of the depth-augmented training image comprises:

cropping the given image from the training image based on the clustering of the pixels of the depth-augmented training image.

6. The method of claim 1, wherein determining the current cluster centers based on the current values of the categorization neural network parameters comprises:

obtaining the current cluster centers from a memory unit of the categorization neural network.

7. The method of claim 1, wherein the clustering loss comprises:

a minimum over each current cluster center of a difference between the current cluster center and the current embedding of the given image.

8. The method of claim 1, wherein the clustering loss further comprises:

a measure of how evenly given images are distributed between the current cluster centers.

9. The method of claim 1, wherein determining the clustering of the target images into the plurality of semantic categories using the final embeddings of the target images and the final cluster centers comprises:

for each target image, assigning the target image to a closest final cluster center to the final embedding of the target image; and

for each final cluster center, determining the target images assigned to the final cluster center as belonging to a same semantic category.

10. The method of claim 1, wherein the categorization neural network comprises a plurality of convolutional neural network layers.

11. The method of claim 1, wherein the current embedding of the given image is an intermediate output of the categorization neural network.

12. The method of claim 1 further comprising, at each of a plurality of training iterations:

processing a selected image depicting a background region using the categorization neural network in accordance with current values of categorization neural network parameters to generate an output comprising a current prediction for whether the selected image depicts an object or a background region;

determining a gradient of an objective function that includes the classification loss; and

determining an update to the current values of the categorization neural network parameters from the gradient.

13. A system, comprising:

one or more computers;

a memory in data communication with the one or more computers and storing instructions that cause the one or more computers to perform operations comprising;

training a categorization neural network to determine trained values of the categorization neural network parameters from initial values of the categorization neural network parameters, comprising, at each of a plurality of iterations;

processing a given image depicting an object using the categorization neural network in accordance with current values of categorization neural network parameters to generate an output comprising;

(i) a current prediction for whether the given image depicts an object or a background region, and (ii) a current embedding of the given image;

determining a plurality of current cluster centers based on the current values of the categorization neural network parameters, wherein each cluster center represents a respective semantic category;

determining a gradient of an objective function that includes a classification loss and a clustering loss, wherein the classification loss depends on the current prediction for whether the given image depicts an object or a background region, and wherein the clustering loss depends on a similarity between the current embedding of the given image and the current cluster centers; and

determining an update to the current values of the categorization neural network parameters from the gradient;

determining a plurality of final cluster centers based on the trained values of the categorization neural network parameters;

for each of a plurality of target images, processing the target image using the categorization neural network in accordance with the trained values of the categorization neural network parameters to generate a final embedding of the target image; and

determining a clustering of the target images into a plurality of semantic categories using the final embeddings of the target images and the final cluster centers.

14. The system of claim 13, wherein the given image is generated by a plurality of operations comprising:

generating a depth-augmented training image by determining a depth associated with each pixel in a training image;

clustering the pixels of the depth-augmented training image using;

(i) the intensity data associated with the pixels of the training image, and (ii) the depths of the pixels of the training image; and

generating the given image based on the clustering of the pixels of the depth-augmented training image.

15. The system of claim 14, wherein determining a depth associated with each pixel in the training image comprises:

processing the training image by using a depth estimation neural network in accordance with trained values of depth estimation neural network parameters to generate an output comprising a depth associated with each pixel in the training image.

16. The method of claim 15, wherein the depth estimation neural network is trained using an unsupervised machine learning training technique.

17. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

training a categorization neural network to determine trained values of the categorization neural network parameters from initial values of the categorization neural network parameters, comprising, at each of a plurality of iterations;

processing a given image depicting an object using the categorization neural network in accordance with current values of categorization neural network parameters to generate an output comprising;

(i) a current prediction for whether the given image depicts an object or a background region, and (ii) a current embedding of the given image;

determining a plurality of current cluster centers based on the current values of the categorization neural network parameters, wherein each cluster center represents a respective semantic category;

determining a gradient of an objective function that includes a classification loss and a clustering loss, wherein the classification loss depends on the current prediction for whether the given image depicts an object or a background region, and wherein the clustering loss depends on a similarity between the current embedding of the given image and the current cluster centers; and

determining an update to the current values of the categorization neural network parameters from the gradient;

determining a plurality of final cluster centers based on the trained values of the categorization neural network parameters;

for each of a plurality of target images, processing the target image using the categorization neural network in accordance with the trained values of the categorization neural network parameters to generate a final embedding of the target image; and

determining a clustering of the target images into a plurality of semantic categories using the final embeddings of the target images and the final cluster centers.

18. The non-transitory computer storage media of claim 17, wherein the given image is generated by a plurality of operations comprising:

generating a depth-augmented training image by determining a depth associated with each pixel in a training image;

clustering the pixels of the depth-augmented training image using;

(i) the intensity data associated with the pixels of the training image, and (ii) the depths of the pixels of the training image; and

generating the given image based on the clustering of the pixels of the depth-augmented training image.

19. The non-transitory computer storage media of claim 18, wherein determining a depth associated with each pixel in the training image comprises:

processing the training image by using a depth estimation neural network in accordance with trained values of depth estimation neural network parameters to generate an output comprising a depth associated with each pixel in the training image.

20. The non-transitory computer storage media of claim 19, wherein the depth estimation neural network is trained using an unsupervised machine learning training technique.