Learning method and learning device for segmenting an image having one or more lanes by using embedding loss to support collaboration with HD maps required to satisfy level 4 of autonomous vehicles and softmax loss, and testing method and testing device using the same

• US 10,311,578 B1
• Filed: 01/23/2019
• Issued: 06/04/2019
• Est. Priority Date: 01/23/2019
• Status: Active Grant

• Current Assignee: Stradvision, Inc.
• Original Assignee: Stradvision, Inc.
• Inventors: Kim, Kye-Hyeon, Kim, Yongjoong, Kim, Insu, Kim, Hak-Kyoung, Nam, Woonhyun, Boo, SukHoon, Sung, Myungchul, Yeo, Donghun, Ryu, Wooju, Jang, Taewoong, Jeong, Kyungjoong, Je, Hongmo, Cho, Hojin
• Primary Examiner(s): Seth, Manav
• Application Number:
  US16/255,339
• Time in Patent Office: 132 Days
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: Y02T 10/40 Engine management systems

Abstract

A learning method for segmenting an image having one or more lanes is provided to be used for supporting collaboration with HD maps required to satisfy level 4 of autonomous vehicles. The learning method includes steps of: a learning device instructing a CNN module (a) to apply convolution operations to the image, thereby generating a feature map, and apply deconvolution operations thereto, thereby generating segmentation scores of each of pixels on the image; (b) to apply Softmax operations to the segmentation scores, thereby generating Softmax scores; and (c) to (I) apply multinomial logistic loss operations and pixel embedding operations to the Softmax scores, thereby generating Softmax losses and embedding losses, where the embedding losses is used to increase inter-lane differences among averages of the segmentation scores and decrease intra-lane variances among the segmentation scores, in learning parameters of the CNN module, and (II) backpropagate the Softmax and the embedding losses.

Claims

1. A learning method for segmenting an input image having one or more lanes, comprising steps of:

(a) a learning device, if the input image is acquired, instructing a convolutional neural network (CNN) module to apply at least one convolution operation to the input image to thereby generate a feature map and then apply at least one deconvolution operation to the feature map to thereby generate each of segmentation scores of each of pixels on the input image;

(b) the learning device instructing the CNN module to apply at least one Softmax operation to each of the segmentation scores to thereby generate each of Softmax scores; and

(c) the learning device instructing the CNN module to (I) (i) apply at least one multinomial logistic loss operation to each of the Softmax scores to thereby generate each of Softmax losses and (ii) apply at least one pixel embedding operation to each of the Softmax scores to thereby generate each of embedding losses which causes a learning of the CNN module to increase each of inter-lane differences among respective averages of the segmentation scores of the respective lanes and decrease each of intra-lane variances among the segmentation scores of the respective lanes, and then (II) learn at least one parameter of the CNN module through backpropagation by using each of the Softmax losses and each of the embedding losses.

2. The learning method of claim 1, wherein each of the embedding losses is calculated by using a formula:

3. The learning method of claim 1, wherein each of the multinomial logistic losses is calculated by using an equation:

4. A testing method for segmenting a test image having one or more lanes, comprising steps of:

(a) on condition that a learning device (1) has instructed a convolutional neural network (CNN) module to apply at least one convolution operation to a training image to thereby generate a feature map for training and then apply at least one deconvolution operation to the feature map for training to thereby generate each of segmentation scores for training of each of pixels on the training image;

(2) has instructed the CNN module to apply at least one Softmax operation to each of the segmentation scores for training to thereby generate each of Softmax scores for training; and

(3) has instructed the CNN module to (I) (i) apply at least one multinomial logistic loss operation to each of the Softmax scores for training to thereby generate each of Softmax losses and (ii) apply at least one pixel embedding operation to each of the Softmax scores for training to thereby generate each of embedding losses which causes a learning of the CNN module to increase each of inter-lane differences among respective averages of the segmentation scores for training of the respective lanes and decrease each of intra-lane variances among the segmentation scores for training of the respective lanes, and then (II) learn at least one parameter of the CNN module through backpropagation by using each of the Softmax losses and each of the embedding losses, a testing device, if the test image is acquired, instructing the CNN module to apply the convolution operation to the test image to thereby generate a feature map for testing and then apply the deconvolution operation to the feature map for testing to thereby generate each of segmentation scores for testing of each of pixels on the test image; and

(b) the testing device instructing the CNN module to apply the Softmax operation to each of the segmentation scores for testing to thereby generate each of Softmax scores for testing.

5. A learning device for segmenting an input image having one or more lanes, comprising:

at least one memory that stores instructions; and

at least one processor configured to execute the instructions to;

perform processes of (I) instructing a convolutional neural network (CNN) module to apply at least one convolution operation to the input image to thereby generate a feature map and then apply at least one deconvolution operation to the feature map to thereby generate each of segmentation scores of each of pixels on the input image, (II) instructing the CNN module to apply at least one Softmax operation to each of the segmentation scores to thereby generate each of Softmax scores, and (III) instructing the CNN module to (1) (i) apply at least one multinomial logistic loss operation to each of the Softmax scores to thereby generate each of Softmax losses and (ii) apply at least one pixel embedding operation to each of the Softmax scores to thereby generate each of embedding losses which causes a learning of the CNN module to increase each of inter-lane differences among respective averages of the segmentation scores of the respective lanes and decrease each of intra-lane variances among the segmentation scores of the respective lanes, and then (2) learn at least one parameter of the CNN module through backpropagation by using each of the Softmax losses and each of the embedding losses.

6. The learning device of claim 5, wherein each of the embedding losses is calculated by using a formula:

7. The learning device of claim 5, wherein each of the multinomial logistic losses is calculated by using an equation:

8. A testing method for segmenting a test image having one or more lanes, comprising:

at least one memory that stores instructions; and

at least one processor, on condition that a learning device (1) has instructed a convolutional neural network (CNN) module to apply at least one convolution operation to a training image to thereby generate a feature map for training and then apply at least one deconvolution operation to the feature map for training to thereby generate each of segmentation scores for training of each of pixels on the training image, (2) has instructed the CNN module to apply at least one Softmax operation to each of the segmentation scores for training to thereby generate each of Softmax scores for training, and (3) has instructed the CNN module to (3-1) (i) apply at least one multinomial logistic loss operation to each of the Softmax scores for training to thereby generate each of Softmax losses and (ii) apply at least one pixel embedding operation to each of the Softmax scores for training to thereby generate each of embedding losses which causes a learning of the CNN module to increase each of inter-lane differences among respective averages of the segmentation scores for training of the respective lanes and decrease each of intra-lane variances among the segmentation scores for training of the respective lanes, and then (3-2) learn at least one parameter of the CNN module through backpropagation by using each of the Softmax losses and each of the embedding losses;

configured to execute the instructions to;

perform processes of (I) instructing the CNN module to apply the convolution operation to the test image to thereby generate a feature map for testing and then apply the deconvolution operation to the feature map for testing to thereby generate each of segmentation scores for testing of each of pixels on the test image, and (II) instructing the CNN module to apply the Softmax operation to each of the segmentation scores for testing to thereby generate each of Softmax scores for testing.