SEMANTIC SEGMENTATION MODEL TRAINING METHODS AND APPARATUSES, ELECTRONIC DEVICES, AND STORAGE MEDIA

• US 20200134375A1
• Filed: 12/25/2019
• Published: 04/30/2020
• Est. Priority Date: 08/01/2017
• Status: Active Grant

• Current Assignee: Beijing SenseTime Technology Development Co., Ltd.
• Original Assignee: Beijing SenseTime Technology Development Co., Ltd.
• Inventors: ZHAN, Xiaohang, LIU, Ziwei, LUO, Ping, LOY, Chen Change, TANG, Xiaoou
• Granted Patent: US 11,301,719 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing

Abstract

A semantic segmentation model training method includes: performing, by a semantic segmentation model, image semantic segmentation on at least one unlabeled image to obtain a preliminary semantic segmentation result as the category of the unlabeled image; obtaining, by a convolutional neural network based on the category of the at least one unlabeled image and the category of at least one labeled image, sub-images respectively corresponding to the at least two images and features corresponding to the sub-images, where the at least two images comprise the at least one unlabeled image and the at least one labeled image, and the at least two sub-images carry the categories of the corresponding images; and training the semantic segmentation model on the basis of the categories of the at least two sub-images and feature distances between the at least two sub-images.

Claims

1. A semantic segmentation model training method, comprising:

performing, by a semantic segmentation model, image semantic segmentation on at least one unlabeled image to obtain a preliminary semantic segmentation result as a category of the at least one unlabeled image;

obtaining, by a convolutional neural network based on the category of the at least one unlabeled image and a category of at least one labeled image, sub-images respectively corresponding to at least two images and features corresponding to the sub-images, wherein the at least two images comprise the at least one unlabeled image and the at least one labeled image, and the at least two sub-images carry the categories of the corresponding images; and

training the semantic segmentation model on the basis of the categories of the at least two sub-images and feature distances between the at least two sub-images.

2. The method according to claim 1, wherein the training the semantic segmentation model on the basis of the categories of the at least two sub-images and feature distances between the at least two sub-images comprises:

establishing a patch graph according to category relations between the sub-images, wherein the patch graph comprises nodes and an edge, wherein the nodes comprise the sub-images, and the edge comprises a feature distance between any two sub-images; and

training the semantic segmentation model to enable the feature distance between two sub-images of a same category in the patch graph to be lower than a first preset value, and the feature distance between two sub-images of different categories to be greater than a second preset value.

3. The method according to claim 2, wherein the establishing a patch graph according to category relations between the sub-images comprises:

selecting at least one sub-image as a reference node, and for at least one reference node;

using a sub-image of a same category as the reference node as a positively correlated node, using a sub-image of a different category from the reference node as a negatively correlated node, and separately establishing a positive correlation connection between the reference node and at least one positively correlated node, and a negative correlation connection between the reference node and at least one negatively correlated node; and

forming a sparse connectivity patch graph according to the at least one reference node, the positively correlated node of the reference node, the negatively correlated node of the reference node, the positive correlation connection, and the negative correlation connection.

4. The method according to claim 2, wherein the training the semantic segmentation model comprises:

training the semantic segmentation model by a gradient back propagation algorithm, so as to minimize an error of the convolutional neural network, wherein the error is a triplet loss of the features of the corresponding sub-images obtained based on the convolutional neural network.

5. The method according to claim 4, wherein the training the semantic segmentation model by a gradient back propagation algorithm comprises:

calculating a maximum error by a loss function according to the feature distance between the sub-images in the established patch graph;

performing gradient back propagation on the maximum error to calculate an error of at least one layer in the convolutional neural network;

calculating a gradient of parameters of at least one layer according to the error of the at least one layer, and modifying the parameters of a corresponding layer in the convolutional neural network according to the gradient;

calculating an error according to the distance between sub-images outputted by the convolutional neural network with the optimized parameters, and taking the error as the maximum error;

iteratively implementing following operations until the maximum error is lower than or equal to a preset value;

performing gradient back propagation on the maximum error to calculate an error of at least one layer in the convolutional neural network; and

calculating the gradient of parameters of at least one layer according to the error of the at least one layer, and modifying the parameters of the corresponding layer in the convolutional neural network according to the gradient.

6. The method according to claim 4, wherein the training the semantic segmentation model comprises:

obtaining parameters of the convolutional neural network based on a training result of the convolutional neural network; and

initializing parameters in the semantic segmentation model based on the obtained parameters of the convolutional neural network.

7. The method according to claim 1, wherein the obtaining, by a convolutional neural network based on the category of the at least one unlabeled image and a category of at least one labeled image, sub-images respectively corresponding to at least two images and features corresponding to the sub-images comprises:

in response to movement of a select box with a preset size on the at least two images, performing determination on pixels in the select box, and when a proportion of pixels of a same category among the pixels in the select box is greater than or equal to a preset value, outputting the image in the select box as a sub-image, and labeling the sub-image as said category; and

obtaining features corresponding to the sub-images by the convolutional neural network.

8. The method according to claim 7, further comprising:

when the proportion of the pixels of the same category among the pixels in the select box is less than the preset value, discarding the select box.

9. The method according to claim 7, wherein the obtaining features corresponding to the sub-images by the convolutional neural network comprises:

performing feature extraction respectively on the unlabeled image and the labeled image by the convolutional neural network, so as to obtain features maps respectively corresponding to the unlabeled image and the labeled image; and

obtaining, based on a position and size of the select box corresponding to the sub-image, the corresponding feature in the select box from the corresponding feature map, so as to determine the features corresponding to the sub-image.

10. The method according to claim 1, further comprising:

before performing, by the semantic segmentation model, image semantic segmentation on the at least one unlabeled image,

using stochastic gradient descent to train the semantic segmentation model until a preset convergence condition is satisfied.

11. The method according to claim 1, further comprising:

before obtaining, by the convolutional neural network based on the category of the at least one unlabeled image and a category of the at least one labeled image, the sub-images respectively corresponding to the at least two images and the features corresponding to the sub-images,

using stochastic gradient descent to train the convolutional neural network until a preset convergence condition is satisfied.

12. A semantic segmentation model training apparatus, comprising:

a memory storing processor-executable instructions; and

a processor arranged to execute the stored processor-executable instructions to perform operations of;

performing, by a semantic segmentation model, image semantic segmentation on at least one unlabeled image to obtain a preliminary semantic segmentation result as a category of the at least one unlabeled image;

obtaining, by a convolutional neural network based on the category of the at least one unlabeled image and a category of at least one labeled image, sub-images respectively corresponding to at least two images and features corresponding to the sub-images, wherein the at least two images comprise the at least one unlabeled image and the at least one labeled image, and the at least two sub-images carry the categories of the corresponding images; and

training the semantic segmentation model on the basis of the categories of the at least two sub-images and feature distances between the at least two sub-images.

13. The apparatus according to claim 12, wherein the training the semantic segmentation model on the basis of the categories of the at least two sub-images and feature distances between the at least two sub-images comprises:

establishing a patch graph according to category relations between the sub-images, wherein the patch graph comprises nodes and an edge, wherein the nodes comprise the sub-images, and the edge comprises a feature distance between any two sub-images; and

training the semantic segmentation model to enable the feature distance between two sub-images of a same category in the patch graph to be lower than a first preset value, and the feature distance between two sub-images of different categories to be greater than a second preset value.

14. The apparatus according to claim 13, wherein the establishing a patch graph according to category relations between the sub-images comprises:

selecting at least one sub-image as a reference node, and for at least one reference node;

using a sub-image of a same category as the reference node as a positively correlated node, using a sub-image of a different category from the reference node as a negatively correlated node, and separately establishing a positive correlation connection between the reference node and at least one positively correlated node, and a negative correlation connection between the reference node and at least one negatively correlated node; and

forming a sparse connectivity patch graph according to the at least one reference node, the positively correlated node of the reference node, the negatively correlated node of the reference node, the positive correlation connection, and the negative correlation connection.

15. The apparatus according to claim 13, wherein the training the semantic segmentation model comprises:

training the semantic segmentation model by a gradient back propagation algorithm, so as to minimize an error of the convolutional neural network, wherein the error is a triplet loss of the features of the corresponding sub-images obtained based on the convolutional neural network.

16. The apparatus according to claim 15, wherein the training the semantic segmentation model by a gradient back propagation algorithm comprises:

calculating a maximum error by a loss function according to the feature distance between the sub-images in the established patch graph;

performing gradient back propagation on the maximum error to calculate an error of at least one layer in the convolutional neural network;

calculating a gradient of parameters of at least one layer according to the error of the at least one layer, and modify the parameters of a corresponding layer in the convolutional neural network according to the gradient;

calculating an error according to the distance between sub-images outputted by the convolutional neural network with the optimized parameters, and take the error as the maximum error;

iteratively implementing following operations until the maximum error is lower than or equal to a preset value;

performing gradient back propagation on the maximum error to calculate an error of at least one layer in the convolutional neural network; and

calculating the gradient of parameters of at least one layer according to the error of the at least one layer, and modify the parameters of the corresponding layer in the convolutional neural network according to the gradient.

17. The apparatus according to claim 15, wherein the training the semantic segmentation model comprises:

obtaining parameters of the convolutional neural network based on a training result of the convolutional neural network; and

initializing parameters in the semantic segmentation model based on the obtained parameters of the convolutional neural network.

18. The apparatus according to claim 12, wherein the obtaining, by a convolutional neural network based on the category of the at least one unlabeled image and a category of at least one labeled image, sub-images respectively corresponding to at least two images and features corresponding to the sub-images comprises:

in response to movement of a select box with a preset size on the at least two images, performing determination on pixels in the select box, and when the proportion of pixels of the same category among the pixels in the select box is greater than or equal to a preset value, outputting the image in the select box as a sub-image, and labeling the sub-image as said category; and

obtaining features corresponding to the sub-images by the convolutional neural network.

19. The apparatus according to claim 18, wherein the processor is arranged to execute the stored processor-executable instructions to further perform an operation of:

when the proportion of the pixels of the same category among the pixels in the select box is less than the preset value, discarding the select box.

20. A non-transitory computer storage medium having stored thereon computer-readable instructions that, when executed by a processor, cause the processor to implement operations of a semantic segmentation model training method, the method comprising:

performing, by a semantic segmentation model, image semantic segmentation on at least one unlabeled image to obtain a preliminary semantic segmentation result as a category of the at least one unlabeled image;

obtaining, by a convolutional neural network based on the category of the at least one unlabeled image and a category of at least one labeled image, sub-images respectively corresponding to at least two images and features corresponding to the sub-images, wherein the at least two images comprise the at least one unlabeled image and the at least one labeled image, and the at least two sub-images carry the categories of the corresponding images; and

training the semantic segmentation model on the basis of the categories of the at least two sub-images and feature distances between the at least two sub-images.