System and method for semantic segmentation using dense upsampling convolution (DUC)

• US 9,953,236 B1
• Filed: 03/10/2017
• Issued: 04/24/2018
• Est. Priority Date: 03/10/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: Tusimple
• Inventors: Huang, Zehua, Chen, Pengfei, Wang, Panqu
• Primary Examiner(s): Brushaber, Frederick M
• Application Number:
  US15/456,219
• Time in Patent Office: 410 Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

A system and method for semantic segmentation using dense upsampling convolution (DUC) are disclosed. A particular embodiment includes: receiving an input image; producing a feature map from the input image; performing a convolution operation on the feature map and reshape the feature map to produce a label map; dividing the label map into equal subparts, which have the same height and width as the feature map; stacking the subparts of the label map to produce a whole label map; and applying a convolution operation directly between the feature map and the whole label map without inserting extra values in deconvolutional layers to produce a semantic label map.

Claims

1. A system comprising:

a data processor; and

an image processing module, executable by the data processor, the image processing module being configured to perform semantic segmentation using a dense upsampling convolution (DUC) operation, the DUC operation being configured to;

receive an input image;

produce a feature map from the input image;

perform a convolution operation on the feature map and reshape the feature map to produce a label map;

divide the label map into equal subparts, which have the same height and width as the feature map;

stack the subparts of the label map to produce a whole label map; and

apply a convolution operation directly between the feature map and the whole label map without inserting extra values in deconvolutional layers to produce a semantic label map.

2. The system of claim 1 wherein the DUC operation is machine learnable.

3. The system of claim 1 wherein the DUC operation is configured to learn an array of upscaling filters to upscale the feature map into a final dense feature map of a desired size.

4. The system of claim 1 wherein the DUC operation is configured operate within a fully convolutional network (FCN).

5. The system of claim 1 wherein the DUC operation is performed at an original resolution, thereby enabling pixel-level decoding.

6. The system of claim 1 wherein the semantic label map is used by an autonomous control subsystem to control a vehicle without a driver.

7. A method comprising:

receiving an input image;

producing a feature map from the input image;

performing a convolution operation on the feature map and reshaping the feature map to produce a label map;

dividing the label map into equal subparts, which have the same height and width as the feature map;

stacking the subparts of the label map to produce a whole label map; and

applying a convolution operation directly between the feature map and the whole label map without inserting extra values in deconvolutional layers to produce a semantic label map.

8. The method of claim 7 wherein the method enables machine learning.

9. The method of claim 7 including learning an array of upscaling filters to upscale the feature map into a final dense feature map of a desired size.

10. The method of claim 7 wherein the method is configured operate within a fully convolutional network (FCN).

11. The method of claim 7 wherein the method is performed at an original resolution, thereby enabling pixel-level decoding.

12. The method of claim 7 wherein the semantic label map is used by an autonomous control subsystem to control a vehicle without a driver.

13. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

receive an input image;

produce a feature map from the input image;

perform a convolution operation on the feature map and reshape the feature map to produce a label map;

divide the label map into equal subparts, which have the same height and width as the feature map;

stack the subparts of the label map to produce a whole label map; and

apply a convolution operation directly between the feature map and the whole label map without inserting extra values in deconvolutional layers to produce a semantic label map.

14. The non-transitory machine-useable storage medium of claim 13 wherein the instructions are further configured to enable machine learning.

15. The non-transitory machine-useable storage medium of claim 13 wherein the instructions are further configured to learn an array of upscaling filters to upscale the feature map into a final dense feature map of a desired size.

16. The non-transitory machine-useable storage medium of claim 13 wherein the instructions are further configured to operate within a fully convolutional network (FCN).

17. The non-transitory machine-useable storage medium of claim 13 wherein the instructions are further configured to perform at an original resolution, thereby enabling pixel-level decoding.

18. The non-transitory machine-useable storage medium of claim 13 wherein the semantic label map is used by an autonomous control subsystem to control a vehicle without a driver.